Tag Archives: Infrastructure

Stability of e-mail verification strongly improved

The e-mail verification on the CAcert web server has recently led to repeated support requests. An analysis of the log files in our data center showed that the corresponding error occurred more frequently. So we have to conclude that many CAcert users have been negatively affected. In order to avoid further negative effects, an emergency
patch was deemed necessary by the Critical System Administrator Team.

The standardised review and testing of the emergency patch implemented yesterday is carried out by the regular teams in the aftermath, which can result in a formal blessing for this patch or a request for additional code or configuration changes. We would like to thank the Critical System Administrator Team for their quick and decisive action. All teams consist of volunteers. If you want to support the work done by the Critical System Administration Team and the review by the Software Team, please donate, to continue to run this service. Thank you.

Updated: Information about Heartbleed-bug in OpenSSL 1.0.1 up to 1.0.1f

German version below

There is news about a bug in OpenSSL that may allow an attacker to leak arbitrary information from any process using OpenSSL.

Good news:

Certificates issued by CAcert are not broken and our central systems did not leak your keys.

Bad news:

Even then you may be affected.
Although your keys were not leaked by CAcert your keys on your own infrastructure systems might have been compromised if you were or are running a vulnerable version of OpenSSL.

To elaborate on this:

The central systems of CAcert and our root certificates are not affected by this issue. Regrettably some of our infrastructure systems were affected by the bug. We are working to fix them and already completed work for the most critical ones. If you logged into those systems, within the last two years, (see list below) you might be affected!
But unfortunately given the nature of this bug we have to assume that the certificates of our members may be affected, if they were used in an environment with a publically accessable OpenSSL connection (e.g. Apache webserver, mail server, Jabber-Server, …). The bug has been open in OpenSSL for two years – from December 2011 and was introduced in stable releases starting with OpenSSL 1.0.1.
When an attacker can reach a vulnerable service he can abuse the TLS heartbeat extension to retrieve arbitrary chunks of memory by exploiting a missing bounds check. This can lead to disclosure of your private keys, resident session keys and other key material as well as all volatile memory contents of the server process like passwords, transmitted user data (e.g. web content) as well as other potentially confidential information.
Exploiting this bug does not leave any noticeable traces, thus for any system which is (or has been) running a vulnerable version of OpenSSL you must assume that at least your used server keys are compromised and therefore must be replaced by newly generated ones. Simply renewing existing certificates is not sufficient! – Please generate NEW keys with at least 2048 bit RSA or stronger!
As mentioned above this bug can be used to leak passwords and thus you should consider changing your login credentials to potentially compromised systems as well as any other system where those credentials might have been used as soon as possible.
An (incomplete) list of commonly used software which include or link to OpenSSL can be found at related apps.

What to do?

  • First ensure that you upgrade your system to a fixed OpenSSL version (1.0.1g or above).
  • Only then create new keys for your certificates.
  • Check what services you have used that may have been affected within the last two years.
  • Wait until you think that those environments got fixed.
  • Then (and only then) change your credentials for those services. If you do it too early, i.e. before the sites got fixed, your data may be leaked, again. So be careful when you do this.

What we are doing:

  • We are updating the affected infrastructure systems and and create new certificates for them.
  • We use this opportunity to upgrade to 4096 bit RSA keys signed with SHA-512. The new fingerprints can be found below. ­čśë
  • We will contact all members, who had active server certificates within the last two years.
  • We will keep you updated, here.

Press release



There is a website where one can check if a domain is affected:
http://filippo.io/Heartbleed/ (Use on your own risk)
We checked our systems against it, it looks like most of the systems we classified as affected are not actually affected. But we decided to update them, anyway and provide them with new certificates, as well.

Status of CAcert Infrastructure systems:

Not affected / Nicht betroffen:

  • main website (www.cacert.org)
  • certificate signer (the system doing the actual certificate issuing)
  • email system (inbound and outbound mail servers)


SHA1 Fingerprint=7A:13:19:98:1E:FB:9F:F9:9A:E6:3D:E5:7D:F0:42:E1:BE:56:B9:79

SHA1 Fingerprint=9B:30:44:3D:A8:8C:F5:5E:50:07:68:70:D6:A1:83:44:F9:7D:00:D6

SHA1 Fingerprint=E1:2C:12:7F:66:DF:2E:9D:F6:BC:FB:6F:BC:F1:2E:A0:10:5F:8E:BA

SHA1 Fingerprint=2E:0B:08:6E:F1:48:FB:76:69:D6:6D:51:8F:E9:B3:2C:C3:4B:14:25

SHA1 Fingerprint=9B:8E:0A:68:96:E5:C8:E6:E6:8E:D8:10:31:3F:7C:2C:A8:4E:E1:3F

SHA1 Fingerprint=27:AB:9B:90:51:9B:BA:60:51:B3:84:54:FF:C7:09:94:63:86:68:FA

SHA1 Fingerprint=87:47:41:6D:C1:32:C7:22:00:E5:DA:E5:3C:4B:28:A2:2B:8A:F3:E4

SHA1 Fingerprint=57:F4:0F:38:1E:53:D0:83:DC:D2:40:0A:13:98:B7:06:55:EA:A7:19

SHA1 Fingerprint=82:33:D3:AE:32:56:C5:AD:9E:BF:D1:84:62:56:EA:95:31:7E:64:8C

SHA1 Fingerprint=6A:AE:16:90:A2:1F:CC:1B:B7:93:71:C0:1B:BD:2E:14:68:69:45:EA

SHA1 Fingerprint=D5:00:0A:15:17:04:2F:50:5B:09:3C:DD:B9:0A:57:DD:B3:BE:3D:B4

SHA1 Fingerprint=B7:59:B9:BA:46:64:E2:D4:C8:73:20:50:45:9B:08:5E:2B:DF:D0:1B

SHA1 Fingerprint=87:07:59:30:30:64:27:15:6E:39:C3:66:09:CA:7A:90:7D:2F:32:32

SHA1 Fingerprint=A2:7A:CB:E7:91:0A:ED:7E:63:9F:D1:97:01:96:E9:7B:F0:9E:43:3D


  • Testserver-management-system (you should have not used correct data there)

Deutsche Fassung:
Ein Bug in OpenSSL wurde gefunden, der es einem Angreifer erlaubt beliebige Informationen jedes Prozesses zu erlangen, der OpenSSL nutzt.

Die gute Nachricht:

Die von CAcert ausgestellten Zertifikate sind nicht kaputt und unsere zentralen Systeme waren auch nicht angreifbar und hat keine Schl├╝ssel verraten.

Die schlechte Nachricht:

Dennoch kann jeder betroffen sein!

Um ins Detail zu gehen:

Die zentralen Systeme und die Stammzertifikate von CAcert sind von diesem Problem nicht betroffen. Leider sind einige unserer Infrastruktur-Systeme durch den Fehler betroffen.
Wir arbeiten daran diese zu fixen und haben dies auch schon f├╝r die meisten erledigt. Jeder, der sich auf diese Systeme in den letzten zwei Jahre eingelogt hat kann betroffen sein!
Aufgrund der Art des Fehlers, m├╝ssen wir leider davon ausgehen, dass die Zertifikate unserer Mitglieder betroffen sind, wenn sie sich in eine Umgebung eingelogt haben, die ├╝ber ├Âffentliche OpenSSL-Verbindungen zug├Ąnglich war (z.B. Apache Webserver, Mail Server, Jaber-Server, …). Dieser Fehler war zwei Jahre lang in OpenSSL – seit Dezember 2011 – und kam beginnend mit Version 1.0.1 in die Stabilen Versionen.
Angreifer, die einen verwundbaren Service erreichen, k├Ânnen die TLS-Erweiterung “heartbeat” ausnutzen, um beliebige Abschnitte aus dem Speicher zu erlangen, indem sie eine fehlende Bereichspr├╝fung ausnutzen. Das kann zur Offenlegung von privaten Schl├╝sseln, im Speicher abgelegte Sitzungsschl├╝sseln, sonstige Schl├╝ssel genauso wie jeglicher weiterer Speicherinhalt des Server-Prozesses wie Passw├Ârter oder ├╝bermittelte Benutzerdaten (z.B. Webinhalte) oder andere vertrauliche Informationen f├╝hren.
Die Ausnutzung dieses Fehlers hinterl├Ąsst keine merklichen Spuren. Daher muss f├╝r jedes System, auf dem eine angreifbare Version von OpenSSL l├Ąuft (oder lief), angenommen werden, dass zumindest die verwendeten Server-Zertifikate kompromittiert sind und deswegen durch einen NEU generierte erstetzt werden m├╝ssen. Einfach die alten Zertifikate zu erneuern, reicht nicht aus! – Bitte NEUE Schl├╝ssel mit 2048 Bit RSA oder st├Ąrker generieren!
Wie oben erw├Ąhnt kann dieser Fehler ausgenutzt werden, um Passw├Ârter zu entwenden. Daher sollte jeder ├╝berlegen, alle Zugangsdaten zu m├Âglicherweise betroffenen Systemen und allen Systemen bei denen diese sonst noch verwendet worden sein k├Ânnen, so bald wie m├Âglich auszutauschen.
Eine (unvollst├Ąndige) Liste an weit verbreiteter Software die OpenSSL verwendet kann z.B. unter folgendem Link gefunden werden.

Was ist zu tun?

  • Als erstes m├╝ssen die eigenen Systeme auf eine fehlerbereinigte Version von OpenSSL aktualisiert werden (Version 1.0.1g oder neuer).
  • Danach neue Schl├╝ssel f├╝r die Zertifikate erstellen. Jetzt ist es sicher das zu tun.
  • ├ťberpr├╝fen, welche fremden Dienste in den letzten zwei Jahren besucht worden sind.
  • Warten, bis dort wahrscheinlich der Fehler behoben wurde.
  • Dann (und erst dann) die Logindaten f├╝r diese Dienste erneuern. Vorsicht: Wenn das zu fr├╝h getan wird, also wenn der Dienst noch nicht bereinigt wurde, k├Ânnen die Daten wieder abgegriffen werden.




Es gibt eine Webseite, bei der man seit kurzem checken kann, ob eine Dom├Ąne angreifbar sind:
http://filippo.io/Heartbleed/ (Benutzung auf eigene Gefahr)
Wir haben unsere Systeme dagegen gepr├╝ft und es sieht so aus, als w├Ąren nicht alle, die wir als potentiell angreifbar eingestuft haben tats├Ąchlich angreifbar, aber wir haben uns dennoch daf├╝r entschieden die entsprechenden Systeme aufzur├╝sten und die Zertifikate zu erneuern.

Was wir tun:

  • Wir arbeiten daran, alle Infrastruktur-Systeme auf den neuesten OpenSSL-Stand zu bringen und f├╝r diese neue Zertifikate zu generieren.
  • Wir nutzen diese Gelegenheit, um dabei auf 4096er-Schl├╝ssel, die mit SHA-512 signiert sind aufzur├╝sten. Die neuen Fingerabdr├╝cke k├Ânnen oben gefunden werden. ­čśë
  • Wir werden alle Mitglieder kontaktieren, die in den letzten zwei Jahren aktive Server-Zertifikate benutzt haben.
  • Wir werden neue Informationen an dieser Stelle ver├Âffentlichen.

CAcert server move completed on December 12, 2013

The move of all existing CAcert servers  to a new smaller rack at the current hosting centre has completed, mostly successful, on December 12 at 23:00 UTC. All main services are available again now, but we still have some smaller problems to sort out, mostly due to the switch-over to a new much more compact firewall with a completely new architecture.

So please bear with us while we iron out the remaining problems, and feel free to report any issues you are still encountering.

Please join me in expressing thanks to the team that worked very hard over four hours after an already stressful day to get this major job completed: Stefan Kooman, Mendel Mobach, Martin Simons.

Blog update

The CAcert Blog (incl. its system) has been upgraded to recent software versions. With this also a new Theme had to be created to be compatible with the current version of the blog software.

Please let me know, if anything was broken by the upgrade or anything does not work for you as expected.

Maintenance / Changes infrastructure

[Update: finished.] I will begin with the work now 2012-04-01 00:00 UTC. Infrastructure e.g. email, wiki, blog, lists will be unavailable during the time. The CAcert website along with all critical systems are not “infrastructure” and will not be affected.

I am planning to work on infrastructure again this weekend (2012-03-31) and downtimes may occur. Our host is low on disk space so I need to do some restructuring. A larger downtime may be involved.

Currently, each virtual host has its own logical volume (LVM). They will be consolidated into one. During the move and resizing I think I will not get around shutting down all infrastructure services.

I will send out a notice before I begin the work.
Sorry for any inconvenience.


I am currently continuing the work until 2012-03-26 07:00 UTC. The following services may be effected (the other services have been migrated already):

  • email
  • lists
  • emailout
  • board
  • translingo

I am working on infrastructure systems. During the next 6 hours the following services may be down as they are moved to another server:

* bugs
* irc
* webmail
* wiki
* email
* cats
* lists
* issue
* emailout

The following services have been shut down:

* dupes
* forum
* logging
* paypal
* test2

svn.cacert.org on new host with client certificate authentication now!

Today I finished the migration of svn.cacert.org to a LXC container on our new infrastructure machine. The container is running on Debian Squeeze and supports some nice new features:

Read only access is provided via http://svn.cacert.org/ as it was before.

Besides allowing client certificate authentication for our Subversion repository this is a big step forward as we now have a modern infrastructure machine with a recent operating system distribution.

If you already have a SVN account on svn.cacert.org and want to use the client certificate authentication feature please send a mail to svn-admin (at) cacert (dot) org.

The Big Masterplan to become Audit Ready

Back in January 2010 the former Board decided by Board motion m20100117.3 “No new subroots on current root, plan for new root”. In the discussion a date was scheduled by end of Dec 31, 2010. On my 2nd thought, probably nobody did recognize, what that means, CAcert's Big Masterplan To become Audit Ready (01/2010) to finish all the projects from the bottom left corner at beginning of 2010 to the top right corner by end of the year with the “New Roots and Escrow” (New Roots and Escrow) process running. So this article should bring Audits mistery to light.

Policy Group worked on the last few essential Policies (Policies on Policy Group), that are essential for the Audit. One essential requirement for Audit is to Rollout the CAcert Community Agreement to all the members, so they can decide to continue or to leave the Community. To become “CCA Rollout Ready” (CCA rollout), the running Software needs to be updated. This opens the next problem: by starting 2010, there was no Software Update Process defined, nor documented. But we’re on the lucky side, the Software-Assessment-Project started November last year to fulfill this requirement (Software-Assessment-Project). The task was: To get a repository system controlled by Software-Assessment team, a controlled testserver environment and a documentation system. Currently the team tests the transfer of a test patch to the production system. Involved parties: Software-Assessment Project team, Software-Assessment team and the Critical Sysadmins team.

CAcert's Big Masterplan To become Audit Ready (10/2010)
CAcert’s Big Masterplan To become Audit Ready (10/2010)

In the meantime, another issue pop’d up: the “Thawte points removal” with a deadline of Nov 16th, 2010. We’ve allready posted several blog posts on this topic. So also this is related onto the Software-Assessment-Project progress (Software-Assessment-Project).

The next topic is running Assurer Training Events (ATE) (Assurer Training Events). ATE’s are an essential concept in the Audit over Assurance (RA) business area. To scale a worldwide community, the community has to assist Auditors work in doing Co-Audits over Assurers. The question: How to contact groups of Assurers was answered back in 2009 with the ATE concept. The purpose of ATE is twofolded: first to communicate to the Assurers all the new informations and second to do Co-Audits. As Assurers follows the invitations to the ATEs we can expect, that they are more active in the community. So also from 2009 ATE experiences, we’ve got new resources from the community by contacts on ATEs (Get new resources). So this was the plan for 2010 ATE season, to find more people, who can help on the several tasks and projects that needs to be finished, before the new Roots and Escrow project and also the Audit can be (re-)started. E.g.

Helping CAcert

  • we are searching Infrastructure Admins for the Non-Critical Infrastructure systems, all running on Unix. Familiar with system migrations for the big Infrastructure project to separate Non-Critical from the Critical systems (The big Infrastructure Task). This project is running about 2 years, but currently without progress.
  • we are searching for Software Developers (C++, Python, Java) for the New Software project BirdShack (New Software Project BirdShack), that was started last year, after Auditors review of the Software that concludes: ÔÇ×Serious difficulties in maintaining, improving and securing.ÔÇŁ and ÔÇ×Cannot form conclusion over software.ÔÇŁ, so if the plan to start with the Audit over the old Software fails, we’re close to the 2nd path: BirdShack.
  • we are searching for Audit consultants who can assists in the Audit next step CrowdIt disclosure system (read AGM – Audit Report 2010 – CrowdIt. CrowdIt, as a sort of wordplay on Crowd-Audit). CrowdIt is an emerging disclosure tool (based on the old DRC browser).
  • we are searching people, who can assist us in the funding project (Funding project), that becomes the ground base for the New Roots and Escrow project (New Roots and Escrow) that should be keep tracked by an Auditor, and the re-start of the Audit (Audit over Assurance (RA) 1) and (Audit over Systems (CA) 2).

The New Roots and Escrow Project Relation to Audit

As said before, the New Roots and Escrow Project should be keep tracked by an Auditor. From the experiences back in 2008 on creating New Roots but fail on Roots Escrow, we’re warned to separate the Audit steps of the New Roots and Escrow Project (New Roots and Escrow) and the Audit over Systems (Audit over Systems (CA) 2). Both tasks should be close together.

On the other side, we have to do an Audit over Assurance (Registration Authority, RA) (Audit over Assurance (RA) 1). There is no requirement on bundling the RA Audit and CA Audit as both business areas have their own Policy sets and can be checked separately. This can make our work presumably easier. Easier to get Audit funding for Audit over RA. As Assurance area is closer to be Audit Ready, we can also signal to the Community Audit is back on track. This will probably push the other tasks. With a small budget we probably can double the result by getting new resources, “Hey, there is progress on the overall Audit task” – CAcert is back!

2009 November Community Update

Original Wiki Post 2009 November Update