By: Nelson M. Nones CPIM, Founder, Chairman and President, Geoprise Technologies Corporation

The marketing, sale and distribution of drug products are regulated in most countries, and fully one-quarter of them uphold very robust regulatory regimes. They include the US and all the European Union (EU) member states as well as Argentina, Australia, Canada, Hong Kong, Iceland, Indonesia, Iran, Israel, Japan, Liechtenstein, Malaysia, Mexico, New Zealand, Norway, Singapore, South Africa, South Korea, Switzerland, Taiwan, Thailand, Turkey and Ukraine which collectively account for over 80% of global demand by revenue. The US alone generated about 45% of worldwide pharmaceutical revenues in 2016.

At the opposite end of the spectrum, just over another quarter of the world’s nations lack any regulatory regimes at all. They are concentrated in Africa, Asia, the Caribbean, Central America and the Pacific Islands. All of them are small, poor or isolated, and account for only a sliver of the worldwide pharmaceutical market.

In between, nearly half the world's nations enforce basic laws and regulations within their domestic markets, which together account for less than 20% of global demand. They include the BRIC countries (Brazil, Russia, India and China), of which India and China are also emerging as key exporters to other markets all over the world. 

Documentation and Recordkeeping Mandates 

In addition to setting strict standards for proving the safety, effectiveness and shelf life of medicines through clinical trials and stability tests, and licensing the marketing of proven products within those countries, robust regulation makes it mandatory for manufacturers and distributors to follow good manufacturing practice (GMP) or good distribution practice (GDP) standards, procedures and controls to:

• Protect patient safety;
• Provide quality assurance (conformance to specifications);
• Control the flow of, and fully account for, all quantities of regulated drug products that are procured, received, converted, dispensed, consumed, made, sold, shipped, returned or disposed; and
• Prevent the introduction of falsified or counterfeit products.

GMP and GDP standards, in turn, require manufacturers, contract manufacturing organizations (CMOs) and distributors to keep detailed documentation and records which prove their compliance with these regulations. Authorities have the power to conduct unannounced inspections and can issue warning letters, seek court orders, initiate product recalls and even revoke licenses to operate when records are found to be poor or incomplete, written procedures are inadequate, or systems are not controlled enough to prevent deviations from good practices. They can conduct these inspections not only within their own countries, but also reach beyond their borders to exporters whose products are imported into their domestic markets.

Mandates for Electronic Records and Signatures

Pharmaceutical manufacturers, CMOs and distributors, therefore, must shoulder a significant recordkeeping burden in order to legally serve the vast majority of their market. Although existing regulations do not preclude them from keeping documents and records on paper, a growing number rely on their enterprise resources planning (ERP) systems, manufacturing execution systems (MES), warehouse management systems (WMS), laboratory information management systems (LIMS), computerized maintenance management systems (CMMS) and document management systems (DMS) to do the job electronically.

In the US, electronic records have been regulated since 1997 under the Code of Federal Regulations Title 21 CFR Part 11. This defines criteria which electronic records, and also electronic signatures, must satisfy to be considered trustworthy, reliable, and legally equivalent to paper records. Section 10(e) states:

Persons who use closed systems to create, modify, maintain, or transmit electronic records shall employ procedures and controls designed to ensure the authenticity, integrity, and, when appropriate, the confidentiality of electronic records, and to ensure that the signer cannot readily repudiate the signed record as not genuine. Such procedures and controls shall include the use of secure, computer-generated, time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify, or delete electronic records. Record changes shall not obscure previously recorded information. Such audit trail documentation shall be retained for a period at least as long as that required for the subject electronic records and shall be available for agency review and copying.

Section 30 sets out similar requirements for open systems, along with additional procedures and controls such as document encryption and use of appropriate digital signature standards. An "open system" is an environment, such as an application hosted by a third party, to which access is controlled by parties who are not responsible for the content of the records it holds.

A broadly similar set of regulations under Directive 2003/94/EC (GMP principles and guidelines in respect of medicinal products for human use) and related guidance (Annex 11: Computerised Systems) apply within the EU member states; however because it uses the word “should” instead of “shall”, Annex 11 Section 9 is somewhat more flexible than its US counterpart:

Consideration should be given, based on a risk assessment, to building into the system the creation of a record of all GMP-relevant changes and deletions (a system generated ‘audit trail’). For change or deletion of GMP-relevant data the reason should be documented. Audit trails need to be available and convertible to a generally intelligible form and regularly reviewed.

Outside the US and the EU, another 20 countries have acceded to the Pharmaceutical Inspection Co-operation Scheme (PIC/S) which develops and maintains a GMP guide that is equivalent to EU GMP guidelines. This means there are only two sets of legal requirements which electronic records must satisfy—one for the US market, which accounts for about 45% of global pharmaceutical demand, and the other covering 48 additional markets both inside and outside the EU which account for about 35% of global demand. Manufacturers, CMOs and distributors who never intend to serve the US market would likely opt for the EU GMP guidelines, which only require them to “consider” implementing an audit trail based on a risk assessment; but anyone else—no matter where they are located—must adhere to the strictest rules which might apply and are therefore required to implement an audit trail.

Audit Trail Security Vulnerabilities

Most enterprise-grade ERP, MES, WMS, LIMS, CMMS and DMS software packages, including the GM-X ERP application, provide built-in audit trail capabilities which meet or exceed the regulatory requirements summarized above.

There is a problem, however.

All these audit trails are stored in a relational database management system (RDBMS). Although applications cannot modify them, anyone having sufficient access privileges and a knowledge of structured query language (SQL) can do so using a variety of database administration tools. This will generally include anyone having database administrator privileges.

Using database administration tools, no system controls exist to prevent someone from obscuring or deleting information that has been previously recorded. What’​s more, nearly all ERP, MES, WMS, LIMS, CMMS and DMS software packages store audit trails in unencrypted form. Compared to encrypted records, it is much easier to modify an unencrypted audit trail without leaving a trace. 

In most organizations, administrators gain access to the RDBMS by supplying a valid user ID and password. If these credentials are compromised—by a hacking attack, for example—​an impostor could gain access to the RDBMS and alter its contents surreptitiously.

A wide range of information security practices, two-factor authentication (TFA) mechanisms and hardening techniques are available to prevent unauthorized access and tampering. But no matter how many of these defenses are employed, the best they can do is reduce exposure to security breach and consequential non-compliance risks. They cannot eliminate these risks.

Blockchain technology can.

How Blockchain Technology Keeps Audit Trails Permanent, Tamper-Proof and Verifiable

Operator identities, entries and actions that cause an ERP, MES, WMS, LIMS, CMMS or DMS application to create, modify, or delete electronic records can be captured as individual transactions. Using blockchain technology, new transactions are batched into blocks and cryptographically hashed in a Merkle Tree using a secure hash algorithm (SHA). Next, a block hash is formed from the resulting transaction root (root of the Merkle Tree), current timestamp and the hash of the previous block, a process which inextricably links the block to its immediate predecessor, and forms a chain which is stored in a blockchain database. The timestamp proves that the transactions must have occurred by then; otherwise they could not have been included in the hash.

Even though block hashes are irreversible, each block is self-certifying because its block hash is quickly and completely reproducible from the transaction data, historical timestamp and previous hash values currently stored with the block. To verify the integrity of these data at any moment in time, a reproduction is made and then compared to the current block hash. The block is deemed acceptable only if those values match. 

From this explanation it should be evident that new transactions—even those reflecting the modification or deletion of existing records—can only ever be appended to a blockchain. It is not possible to delete or rewrite existing blocks on the chain because those alterations will always be rejected. As a result, the historical audit trail is always preserved in a permanent, tamper-proof and verifiable way.

Further, unlike conventional enterprise applications, blockchain technology is decentralized and operates over a network. Each node in the network holds its own complete copy of the blockchain database. New transactions propagate to every node. As soon as one of those nodes is able to form and verify a new block from these transactions, the block is automatically broadcast in the form of a message sent to all the other nodes. This process is fault tolerant and self-correcting: if a node fails to receive a message, it will automatically request the missing block from the other nodes whenever the next message arrives.

The node's blockchain server will append an arriving block to its local copy of the blockchain database only after it deems the block to be acceptable. In this way, even if an attacker were able to break into and alter the blockchain database at one of the nodes, the other nodes in the network remain pristine because they will always reject the attacker's alterations.

The decentralized nature and self-certification capabilities of blockchain technology completely eliminate the security vulnerabilities exposed by storing audit trails in a centralized RDBMS. 

For further details and an illustration of these concepts, see pages 4 and 5 of our white paper. You can also read the original blockchain specification here.

The GM-X ERP for Blockchain Solution

​For manufacturers, CMOs and distributors of pharmaceutical products, GM-X ERP for Blockchain is the world's first and only total solution for maintaining electronic documents, records and signatures in full compliance with all applicable US, EU and PIC/S regulations and guidance.

Various other ERP, MES, WMS, LIMS, CMMS and DMS applications may satisfy some of these requirements, but none provide a total solution:

• None of them offer blockchain technology out of the box for securing audit trails.
• ​Other ERP software packages currently on the market do not provide any electronic document management capabilities.
• DMS applications, in turn, do not keep procurement, production, control or distribution records for products.
• CMMS software packages are suitable only for keeping plant maintenance logbooks.
• By definition, MES software does not satisfy any documentation and record keeping requirements beyond the plant floor.
• Also by definition, WMS software does not satisfy any documentation and record keeping requirements beyond the warehouse.
• And finally, by definition, LIMS software does not satisfy any documentation and record keeping requirements beyond the laboratory.  

In the past, pharmaceutical manufacturers and distributors wishing to keep electronic documents and records had to select and deploy separate software packages for each of these functions. Integrating and maintaining all these packages is a time-consuming and costly endeavor.

The total GM-X solution comprises the Alert, Blockchain, Content, Inventory, Order, Menu, Party, Product, Shipment, Survey, Work Effort and Workflow subsystems.

Configuring a System-Generated Audit Trail on the Blockchain

During system implementation, various Blockchain Triggers are configured using the GM-X Blockchain subsystem. Tony Marston explains how to do this in his blog post, Adding Blockchain to an ERP System.

To illustrate how this can be done for electronic records, consider the following requirement which is distilled from US Title 21 CFR Part 211, Subpart J (Records and Reports), Sections 184(c) and 188; and also EU GMP Volume 4, Chapter 4: Documentation, Sections 4.8 and 4.21:

At the time each action is taken, the system must maintain an individual inventory record of each component and packaging material batch and, for each component batch, records of material issued to production, material used in production, material destruction, returns of materials that have not been used in production to storage, and reconciliation of the use of each batch of such component. ​The inventory record must contain sufficient information to allow determination of any batch of drug product associated with the use of each component, ingredient or packaging material (i.e. traceability).

In the GM-X system, the material master is the Product table, and the batch master is the Lot table. All inventories (also known as “quants” in a WMS, representing the stock of any material in one container having the same batch and serial number) are recorded in the Inventory Item table, including components and packaging material.

As with any WMS, staging material from storage to the production area, and returning unused material from the production area to storage, is performed in two steps using GM-X Transfer Orders. The first step creates records in the Item Issuance table when the material is picked up, and the second step creates records in the Item Receipt table when the material is dropped off. Both steps will update the Inventory Item table, but at different times. In between, the inventories are in transit and are tracked by way of the Package table. Creation of each Package will also create Outbound Package Content and Inbound Package Content records which have identical quantities, and are linked to specific Item Issuance and the related Inventory Item records. This makes it possible to reconcile material and batches sent from the pickup point with what was received at the drop-off point.

For material staging and returns, it is also possible to perform one-step material transfers without reference to a Transfer Order in the following ways:

• Transfer some or all of the material in one Container to another Container. This creates records in the Inventory Item Movement table and also updates the Inventory Item table.
• Move an entire Container and all its contents from one Facility to another. This creates records in the Container Movement table and also updates the Inventory Item table.
• When Containers form a nested hierarchy of movable handling units, the Inventory Item table records material stored in Containers at the lowest level of the hierarchy; for example, material packed into shipping cases. When those cases are placed onto pallets, and the pallets are then placed in shipping containers, GM-X records the placements in the Container Movement table but there is no need to update the Inventory Item table because the contents of each shipping case haven’t changed. 

No Item Issuance or Item Receipt records are created for one-step material transfers because no inventories have been withdrawn from or received into stock.

During weighing, dispensing, production and packing operations, GM-X handles the consumption or conversion of components (raw materials, ingredients or semi-finished products such as bulk tablets) and packaging material using Transfer Orders which create records in the Item Issuance table, deduct withdrawn quantities from the Inventory Item table, and also create corresponding records in the Package and Outbound Package Content tables. The Outbound Package Content records are linked to specific Item Issuance and Inventory Item records which identify the materials and batches which have been charged to work in process (WIP).

Upon completion of production, Transfer Orders are also used to receive finished packs and semi-finished products into stock. These create records in the Item Receipt table, add received quantities to the Inventory Item table, and also create corresponding records in the Package and Inbound Package Content tables​. The Inbound Package Content records are linked to specific Item Receipt and Inventory Item records which identify the materials and batches withdrawn from WIP during the receiving process. In addition, records linking produced batches to the batches of components, ingredients and packaging material consumed in production are recorded in the Item Receipt Issuance Link table to provide traceability. This makes it possible to fully reconcile materials and batches issued to production with the use of materials and batches consumed in production.

The foregoing discussion proves that the GM-X ERP application fully satisfies the requirement given above, but how should GM-X be configured to maintain a system-generated audit trail of these entries and actions on the blockchain?

One way is to define the Item Issuance, Item Receipt, Inventory Item Movement and Container Movement tables as Blockchain Triggers.

To maintain a single, comprehensive audit trail, each of these Blockchain Trigger configurations can publish transactions to a single data stream belonging to a single blockchain. Transactions can be distributed to all nodes in the blockchain, either in plain text or encoded (encrypted). Plain text is suitable only for closed systems, but a closed system can be configured to distribute encoded transactions to assure confidentiality. However, if any of the nodes in the blockchain qualifies as an “open system” as per US Title 21 CFR Part 11, Section 30, then the distribution must be encoded.

No Trigger Column and Trigger Value entries are required because every insert, update and deletion affecting the Item Issuance, Item Receipt, Inventory Item Movement and Container Movement tables needs to fire a Blockchain Trigger which adds a transaction to the audit trail.

The illustration below shows an example of Related Tables for the Item Receipt trigger. This configuration only needs to be done once, when the GM-X system is being implemented, and requires very little time. Our recent experience demonstrates that it takes less than two hours for a trained implementer to fully configure all four Blockchain Triggers.

This illustration also clearly shows that the resulting audit trail will tell the whole story behind each and every action. Every Item Issuance transaction will record the quantity issued, the entry timestamp and the user who entered the transaction—plus, through the Related Tables:

• ​Who performed each role required by the action’s standard operating procedure (Organization and Item Issuance Role together with the full name and ID of the Person who performed the action).
• What was issued (Lot details, including the batch expiry date; Product ID, name and unit of measure; Container ID, type and description; and visible Package ID).
• Where the action occurred (Facility ID, name and type).
• When the action occurred (dates in the Package Status History when the Package status changed).
• Why the action was performed and for whom (Sales Order number plus the full name and ID of the external customer Organization; or Transfer Order number plus the full name and ID of the internal receiver Organization).
• How the action was carried out (the Picklist Detail information utilized).

Here is another mandate distilled from US Title 21 CFR Part 211, Subpart J (Records and Reports), Section 188; and also EU GMP Volume 4, Chapter 4: Documentation, Sections 4.20 and 4.21. It requires manufacturers and CMOs to keep batch production records:

​The system must keep batch production records including dates; times (when appropriate); identity of major equipment used; manufactured material number and name; manufactured batch number and quantities; batch numbers, quantities and weights of raw materials, intermediates, or any reprocessed materials used; any sampling performed; in-process and laboratory test results; actual yield at appropriate phases or times; results of release testing; all analytical records relating to the batch (or a reference that will permit their retrieval) and a usage decision, with the date and signature of the qualified person (QP) who made the decision.

Many pharmaceutical manufacturers and CMOs record this information on paper forms known as “travelers” because they move through production along with the manufactured batches. Operators write this information onto the forms and sign them by hand. In the past, they were then filed away in vaults during the required retention period.

Today, many of these organizations have found that scanning these forms, and uploading the scan copies into a DMS, is more expedient and cost-effective than deploying electronic data collection systems in production areas and laboratories. Using the GM-X Alert and Workflow subsystems, they can upload these files into the controlled GM-X document repository and then obtain the QP’s electronic signature. A Blockchain Trigger for the Alert table causes the GM-X Blockchain subsystem to publish these files on the blockchain, together with the document metadata and electronic signatures, thus forming a permanent, tamper-proof and verifiable audit trail of production batch records.

The same Blockchain Trigger configuration can also be used for static documentation such as standard operating procedures and work instructions. 

Accessing and Reading Electronic Records and Signatures on the Blockchain

To facilitate inspections, all US, EU and PIC/S regulations and guidelines require electronic audit trails to be retrievable and readable by agency personnel. Pharmaceutical manufacturers, CMOs and distributions also make frequent use of these audit trails when conducting internal audits and investigations.

The GM-X Blockchain subsystem provides an interactive task for viewing electronic audit trails directly from the blockchain. Because GM-X is a responsive Web application, this task runs in any modern browser installed on a desktop or laptop computer, tablet computer or smartphone. It will automatically decrypt any data and files which are encoded on the blockchain.

The simplest blockchain network has two nodes. The primary node is connected to the GM-X ERP application where users enter transactions and upload documents that are published to the blockchain. The secondary node holds an archive of the audit trail in its copy of the blockchain database. A separate instance of the GM-X ERP application is connected to the secondary node and provides access to the viewing task. It has its own Logon screen and role-based access controls, and requires only the GM-X Audit, Blockchain, Menu and Content subsystems.

With this configuration, only the primary node can publish new transactions to the blockchain. The secondary node can receive new transactions from the primary node, and verify them, but is only permitted to read its copy of the blockchain database. Both nodes connect to one another over the organization’s private local area network (LAN) or corporate wide area network (WAN) to form a closed system.

Alternatively, the secondary node and its instance of the GM-X ERP application can be hosted in the public cloud. This node and its GM-X instance are accessed over the public Internet and form an “open system” which means that transactions and documents must be encoded before they are published to the blockchain.

Nodes can always be added to the blockchain to handle situations which are more complex. For example, many pharmaceutical manufacturers and distributors utilize third-party logistics (3PL) providers and CMOs. If some of the transactions to be recorded in the audit trail originate from third parties, each party can publish transactions at its own node, using its own GM-X instance. Similarly, larger pharmaceutical enterprises can run separate instances of the GM-X ERP application at multiple nodes. No matter where they originate, all transactions will be added to a consolidated copy of the audit trail which will be visible at the secondary node.   

Extra secondary nodes can also be added to address business continuity and disaster recovery requirements. 

GM-X ERP for Blockchain: Your Total Solution for Electronic Records and Signatures

Geoprise Technologies has many years of experience successfully implementing, validating and using ERP, MES, warehouse management, laboratory management and document management systems at numerous pharmaceutical sites of all sizes across four continents—including five of the world’s top 20 pharmaceutical enterprises.

We have applied our industry knowledge and cumulative experience to build GM-X ERP for Blockchain, the world’s first and only total solution for maintaining electronic documents, records and signatures in full compliance with all applicable US, EU and PIC/S regulations and guidance.

GM-X ERP for Blockchain is also the world’s first and only ERP application to incorporate proven blockchain technology out of the box. Blockchain technology eliminates a known security vulnerability affecting conventional electronic audit trails which exposes pharmaceutical manufacturers and distributors to potentially serious non-compliance risks.

Our blockchain solution is already available, requires no complex programming effort or interfaces, and can be configured to properly maintain a full and completely secure audit trail in a matter of hours. By comparison, we know that other software vendors are only beginning to explore blockchain technology, or initiate custom pilot projects for selected clients. It will take considerable time and expense for their efforts to bear fruit. If you can’t afford to wait that long, I invite you to evaluate GM-X ERP for Blockchain now, and the Geoprise team looks forward to serving you in the near future.