Mashup Integration

Integration Mashup (Composites)

• A mashup is a Web page or application that combines data or functionality from two or more sources to create a new service.
• Mashup integration happened at the UI layer not on data or logical layer, so it does not pass the data into two systems.
• Link or Embedded page can be used to bring the external UI in Salesforce

How to use a link to implement Mashup?

• The link can be used to open a window to display external system using
• Custom or Standard buttons, custom links, or formula
• Visualforce page <Apex:outputlink> tag
• Whenever mashup is implemented with link, it always be asynchronous means no effect on page load.

How to use VF page to implement Mashup?

• Visualforce mashups:
• Display data in the salesforce interface that is not physically stored in salesforce
• Whenever mashup is implemented with link, it always be synchronous means user to wait until external content is loaded
• Visualforce mashups are implemented with
• Web tab
• VF iFrame tag

 To know about other integration API at a glance, pls Click here

Integration Patterns

In the real world, no standalone application can fulfill customer need and one system is always rely on other system and hence Salesforce also needs to talk to another system for different purposes. Although each integration scenario is unique, but each must be solved with one kind of patters.

List of Patterns-The following are the patterns that cover all strategies to connect Salesforce with external applications:

1. Request & Reply - Salesforce invokes the process to the remote system, waits for completion and reply, track state based on the response.
2. Fire & Forget- Salesforce invokes a process to a remote system, receives acknowledgment (doesn't wait for completion), and hands-off control back to Salesforce
3. Batch data Synchronization- Data stored in force.com should be created/refreshed to reflect updated by an external system, also changes in force.com data sent to the external system (bi-directional batch).
4. Remote Call-in- Data stored in force.com is created, updated, or deleted by the external system.
5. UI Update Based on Data Changes- Salesforce UI to automatically update based on salesforce data change.
6. Data Virtualization-Salesforce accesses external data in real-time.

Pattern Approach-The integration patterns in this article are classified into three categories:

• Data Integration—These patterns address the requirement to synchronize data that resides in two or more systems so that both systems always contain timely and meaningful data.
• Process Integration—The patterns in this category address the need for a business process to leverage two or more applications to complete its task.
• Virtual Integration—The patterns in this category address the need for a user to view, search, and modify data that are stored in an external system.

Request & Reply –

• Salesforce calling remote system to perform an action waits for the reply synchronously and updates the response back in Salesforce
• Example - Salesforce collection order info and sends to the Order processing system (SAP ECC) and waits for the Order status/number and updates in salesforce.

Forces- 

• Is sync call necessary?
• Do Salesforce need to process the response in same transaction
• Message size (small/large)
• Is UI based event or DML based data change event that triggers this invocation?

Solution:

• Vf page to initiate APEX SOAP callout in sync manner (UI initiated action)
• Salesforce consumes system WSDL and generate Apex proxy classes to call remote service. Vf page calls this proxy class to invoke remote service
• Vf page to initiate APEX HTTP callout in sync manner (UI initiated action)
• RESTful http services will be called by vf pages to invoke remote service

Additional Considerations:

•  Salesforce has a configurable timeout up to 60 seconds for calls from Apex. Completed of apex process should be within this time or proper message should be handled.
• This pattern is primarily for small volume real-time activities.
• If Salesforce is the master, external system should store the Salesforce Id as external key. If Remote system is master, Salesforce should store the external id from the remote sytem

Limits:

• Only 10 callouts per execution context
• Max 60 seconds per callout. Max 120 seconds for all callouts
• Max message size = 3MB (for request/response)

Fire & Forget

• As the name suggests fire the call and forget. Salesforce invokes a process to the external system, receives acknowledgment but doesn't wait for completion and hands-off control back to Salesforce
• Example - Salesforce collection order info and sends to Order processing system (SAP ECC) and lets it handle for Order completion. Also optionally Order status/number is updated in salesforce.

Forces:

• Is sync call necessary?
• Is the message size small?
• Is the integration based on the occurrence of a specific event, such as a button click in the Salesforce user interface, or DML-based events?
• Is guaranteed message delivery from Salesforce to the remote system a requirement?
• Does the endpoint or the Enterprise Service Bus (ESB) support long polling?
• Are declarative configuration methods preferred over custom Apex development? In this case, solutions such as platform events are preferred over Apex callouts.

Solution:

• Platform events- Best
• Outbound messaging
• Calls to a remote system can be performed from a batch job.
• Vf page with async callout Not the best fit. Also, it needs to handle guaranteed delivery by custom code.

Additional Considerations:

• The remote system must handle subsequent errors when the initial invocation is handed off for asynchronous processing.
• Workflow rules can't track deletion of a record, only inserts or update of a record. To send an outbound message for deletion - develop trigger to create new record in custom object for deleted ids. Implement workflow outbound on custom object send delete the message

Calling Mechanisms-

• Process Builder(Used for platform event)
• Lightning component or Visualforce and Apex controllers (Used to invoke a remote process asynchronously using an Apex callout.)
• Workflow rules (Used only for the outbound messaging solution)
• Apex triggers (Used for platform event)

Limits:

• Waits for acknowledgment upto 10 seconds, after 10 seconds salesforce tries to resend message for up to 24 hours. The retry frequency increases exponentially starting at 15 sec interval and ending with 60min interval. Admin can retry manually (monitoring from the queue after 24hr period)
•  Single outbound message can have upto 100 records
• Only 10 call-outs per execution context
• Max 60 seconds per callout . Max 120 seconds for all callouts
• Max message size  = 3MB (for request/response)

Batch data Synchronization

• Extract and transform object data from/to Salesforce with from/to external system for one time or ongoing basis.
• Example - ETL billing data into salesforce from remote system on a weekly basis

Forces:

• Aync Data Integration = Batch
• Should Data stored in salesforce?
• Should data be refreshed in salesforce if remote system changes data (on a scheduled basis?)
• Data support primary biz process/analytics requirements?

Solution:

• 3rd party ETL that uses Bulk API / SOAP API from remote to salesforce for change data capture
• 3rd party ETL that uses Bulk API / SOAP API from Salesforce to remote change data capture

Additional Considerations:

• If multiple database is consolidated to Salesforce, maintain Control tables in an intermediate on-premise database
• Maintain LastRunTime in control table and other needed values from Control table and use it for next execution
•  If processing is successful, update the control values in control table
• If processing failed, update the control vlaues that enable a restart and exit
• If importing child records of Master-detail relationship, group the imported data using its parent key at the source to avoid locking. Ex - If importing contact data for account, group all contacts per account, so you can load in bulk.
• Timeliness - Running batch during normal biz operation might lead to data contention either on user side or batch fail if same record is accessed. Use Data segmentation by record type to minimize the effect or run the batch on non-biz hours if possible

Remote Call-in

• As name suggestion remote application makes the call to salesforce. Invoking Salesforce for updating/querying the Salesforce from remote system
• Example - Order processing system (SAP ECC) update order status in salesforce after processing the order

Forces:

• Is sync call necessary? 
• Do Salesforce need to process the response in same transaction?
• Message size (small/large)?
•  Is transaction processing required?
• If the remote system is SOAP-capable, is the remote system able to participate in a contract-first approach, where Salesforce dictates the contract? This is required where our SOAP API is used, for which a predefined WSDL is supplied.

Solution:

• SOAP API - Generate WSDL (using Partner/Enterprise),  Synchronous API, Query and DML operations, Bulk processing (for 500K or more records use Bulk API)
•  REST API -  Synchronous HTTP api for query and dml ops,
• Apex web services
• Apex REST services
• Bulk API

 Limits:

• Session timeout -
• Query timeout - each query has a individual timeout of 120 seconds.
• runs 200 records / request.
• Default batch size = 500 records (max 2000 records). Use queryMore() for fetching additional records if max reached
• Bulk API
• Max 10k records per batch
• Max 5000 batches / 24 hr period

UI Update Based on Data Change

• Real-time update of CRM data changes in Salesforce UI
• Ex - Call center agent wants to see the payment processing record while on a call with customer when customer pays it to the payment center

Forces

• Does the data being acted on need to be stored in Salesforce?
• Can a custom user interface layer be built for viewing this data?

Solution

• Salesforce Streaming API to update Salesforce UI
• PushTopic with query defn for what data is needed with filter conditions
• Javascript based implementation of Bayeux protocol
• Visualforce page
• JS library in static resource

Additional Considerations

• Delivery and order of notification is not guaranteed
• No notifications for changes made from Bulk 

Data Virtualization

• In Salesforce, how do you view, search, and modify data that’s stored outside of Salesforce, without moving the data from the external system into Salesforce?
• Orders are managed by an external (remote) system. But sales reps want to view and update real-time order information in Salesforce without having to learn or use the external system.

Forces

• Do you want to build a declarative/point-and-click outbound integration or UI mashup in Salesforce?
• Do you have a large amount of data that you don’t want to copy into your Salesforce org?
• Do you need to access small amounts of remote system data at any one time?
• Do you need real-time access to the latest data?
• Do you store your data in the cloud or in a back-office system, but want to display or process that data in your Salesforce org?
• Do you have data residency concerns for storing certain types of data in Salesforce?

Solution-

• Salesforce Connect

To know more about the security aspect of integration, pls Click here

To know about integration API at a glance, pls Click here