Tag Archives: signer

Backlog in the delivery of certificates

Unfortunately, there has been a backlog in the delivery of renewed and new certificates in Signer. The Critical Team cannot solve this remotely. A visit to the data centre is planned for next Monday. The stuck certificates should then be delivered. We are sorry that there has been a disruption again.

Im Signer ist es leider zu einem Stau der Auslieferung erneuerter und neuer Zertifikate gekommen. Das Critical Team kann dies nicht per Fernwartung lösen. Ein Besuch des Rechenzentrums ist am kommenden Montag geplant. Die feststeckenden Zertifikate sollten anschliessend ausgeliefert werden. Es tut uns Leid, dass es erneut zu einer Störung gekommen ist.

New signer proves itself in use

1 Reply

EN: Signer is running again

DE: Signer ist wieder in Betrieb

FR: Signataire fonctionne à nouveau

ES: Firmante vuelve a funcionar

IT: Firmatario è di nuovo in funzione

The signer has been running again since yesterday, Friday, around 13:00 CEST. We then (while we were doing other work) watched the processing for about another hour… Around 0:30 CEST all outstanding certificate requests (~3000) were processed.

Things didn’t quite go as planned in June. As soon as something cannot be done remotely – there is no remote access to critical systems for security reasons – someone who is authorised to do so has to go the data centre in the Netherlands. Despite Corona, quarantine, floods, overtime at the company and whatever else comes up. That’s maybe two hours. Then two hours home again and in between the actual work. During the opening hours of the data centre, in your free time and paying for your own train ticket or petrol. It’s not always easy to reconcile all that. On Friday afternoon, however, the time had come and the Signer has now been running smoothly again for over a day.

As can be seen from the Critical Team’s plan published yesterday, preliminary work is already underway to make the system redundant throughout and even more robust, so that failures should no longer be noticed by users, because no one is interested in such failures! We are very sorry that you had to wait so long. At the same time, we thank the small core team who have sacrificed nights and weekends over the last five weeks to get the technology back up and running for the CAcert community!

Work on new signer in full swing

6 Replies

We recently reported here about a signer malfunction in our data centre in the Netherlands. Usually, our systems are redundant for such cases. This spring, the second disk was removed for repair due to an other malfunction that could not be resolved remotely. A small team of our volunteer engineers is working flat out to rebuild the server locally in Germany.

Then it can be brought to the data centre on the following Friday. If everything goes according to plan by Wednesday, the server in the Netherlands can go online on Friday.

CAcert is run exclusively by volunteers in their spare time. If you would like to help out in one way or another, we recommend subscribing to the relevant mailing lists to get in touch with the current topics and the active players.

Arbeit an neuem Signer läuft auf Hochtouren

Wir haben vor kurzem hier über eine Störung des Signers in unserem Datenzentrum in den Niederlanden berichtet. Üblicherweise sind für solche Fälle unsere Systeme redundant ausgerichtet. In diesem Frühjahr wurde die zweite Platte wegen einer anderen Störung, die sich nur per Fernwartung lösen konnte zur Reparatur entfernt.

Ein kleines Team unserer freiwilligen Ingenieure arbeitet mit Hochdruck daran, den Server lokal in Deutschland nachzubauen. Daraufhin kann er am darauffolgenden Freitag ins Rechenzentrum gebracht werden. Sollte alles bis Mittwoch nach Plan laufen, kann der Server in den Niederlanden am Freitag ins Netz gehen.

CAcert wird ausschliesslich von Freiwilligen in Ihrer Freizeit betrieben. Wenn Sie auf die eine oder andere Art mithelfen möchten, empfehlen wir, die entsprechenden Mailinglisten zu abonnieren, um so mit den aktuellen Themenkreisen und den aktiven Akteuren in Kontakt zu kommen.

New Software: The Heart of Gold

The Heart of Gold

With the last two posts and we talked about why we are rewriting our software and what the main changes on the user side will be. In today’s post we’re going to look behind the scenes to see how the signer “the heart of the system” works.

While looking at the existing Perl code the developing team decided that it is worth to do a code rewrite and refactoring for the signer too. This decision was made after analysis showed that the current code will most likely not be easily adjustable to satisfy our future requirements. Even with the most basic requirements that were layed out for the new software the code was failing several aspects and thus, instead of heavily rewriting things, it makes less work to ignore the existing code base and start anew. Going that way also allowed the team to freely choose the language best suited their needs. So for the signer C++ was chosen as coding language.

One aspect of immediate interest is and will be the new root structure which is currently under development in a separate project called “NRE” (New Root and Escrow). This structure requires the signer side to work with sub roots to sign the certificates. The primary root will be only used to sign the intermediate and sub (“profile”) roots. There will be a set of sub roots for the different kinds of certificates that CAcert is offering. One sub root for email, signing and login client certificates, one for code singing certificates, one for server certificates, one for organisation client certificates, one for organisation server certificates and so on. This also removes the differentiation between different “Classes” as they have never fit the CAcert model perfectly (see also http://serverfault.com/questions/365846/ssl-certificate-class-2-vs-class-3-vs-class-4 ) Instead the use of these certificates will be defined by certificate profiles selected in the front end which are passed onto the signer.

To avoid the problem with large CRL files, we are facing with the existing roots, there is the idea of having all the sub roots signing a set of further sub roots which will be used for a shorter time span of e.g. three to six months so that the CRL files will stay a reasonable size of below 1 MiB. That is also an additional complexity for the signer to handle, as the signer has to choose the correct sub root to sign the current task automatically. Also the management of the different CRLs would be hard with something similar to the current signing software.

Apart from the actual certificates used for signing there is the actual technology to handle all the cryptography. One aspect here is to be able to switch the crypto backend to what we think will be the best one at that time, i.e. OpenSSL, GnuTLS, WolffSSL or whatever provides the security guarantees we require. This will allow CAcert to react quickly if severe problems are found in a crypto backend. Currently only the use of OpenSSL is implemented, but interactions with OpenSSL have been kept together so that in implementation of another crypto backend is feasible.

Being able to switch out parts of the signer software with other libraries provides us with more flexibility at compile time and for reusing the code for testing. Together with the use of unit tests, that are run under Jenkins, there is a good change to see problems in the source code before things hit the life system. Also having the continuous integration aspect with Jenkins it allows us to check that current changes won’t break existing behaviour unless we want to have that behaviour actually changed.

Furthermore a new protocol for the communication between the signing system and the database has been developed. First of all, all communication gets wrapped into TLS to have an encrypted and authenticated connection, where both sides identify themselves to each other. Inside this secure channel a record-based protocol – similar to what is used to flash firmware to microcontrollers and EPROMs – is used to transmit all information required for the signing task. Building on the information received in those records the signer rebuilds the signing request and verifies the information (and their proofs of freshness) to be correct.

After a few further checks the signer hands this data to the crypto backend. This way all user entered data (common name, subject alternative names, organisation name, etc) stays UTF-8 encoded during the whole process. Also, due to the integrated library, there is no need to re-encode user input in specific formats for an external tool to process the signing task.

The protocol also contains several safety checks to make sure that only verifiable and properly embedded information is sent to and processed on the signer. One of them is a “proof” protocol to transmit facts about the state of the data affecting the signing task. On the other hand the protocol has been kept scalable so that it can be easily extended for future needs, e.g. if additional information or commands need to be processed.

For issuing a new certificate the signer client fetches all necessary information from the database. It then opens a connection to the signer and transmits all data for the certificate starting with the CSR or SPKAC-request for the certificate. When all data is transmitted the client asks the signer to sign the current certificate (“sign”). The signer checks the requests and signs it, but instead of returning the resulting certificate, it answers with a log over the current signing session (“setLog”). When the client confirms that the log has been saved (this could be implemented with a hash over the log beeing transmitted back in the “logSaved”-record) the signer transmits the new certificate (“respondCertificate”) and a code name of the sub CA that was actually used to sign this request (“signingCA”). Now the signer client writes the certificate, its serial, issuance and expiry date and possibly more data back to the database and closes this signing session.

For revoking certificates the signer client fetches serials of the same sub-ca (the certificate that actually signed the certificate to be revoked) and sends them ( with “addSerial”) over a fresh connection with the information which CA it is (as argument to “revoke”) to the signer. The signer fetches the current date and time, marks all given certificates as revoked on that instant, resigns the CRL and returns the exact revocation date and time, the X509_Algorithm Structure used to resign the CRL, the actual signature and the “lastUpdate” and “nextUpdate” fields from the CRL (as argument to “revoked”). That data is ASN1 encoded so it can easily be split afterwards. The client updates his local CRL accordingly with the given dates and the given signature and verifies that the signature is valid again. If the updated CRL validates the revocation process has successfully completed and both CRLs (on the signer, and on client) are in sync again. If the CRL validation fails, the client requests the full CRL to be transmitted in order to get the CRLs back in sync. This is not expected to happen in normal production service.

The internal code names for the various parts of the new software are based on the characters of the Michael Ende’s novel MOMO. Based on the story we are calling our web front end Gigi, as – like in the story – it leads your way. Cassiopeia on the other hand is a wise, trusty turtle with a hard shell to protect its secrets and thus the perfect name for our signer.

If you are interested in our work you can find the source code of the new software as well as other CAcert projects mirrored on . To help with development of our software please contact the development mailing list cacert-devel@list.cacert.org.
There will be a live presentation with the new software as well as the possibility to discuss the ideas with the core developers of the new software at the CAcert booth on the CLT2015.

Problems with signing certificates (Solved)