HTTP Response Codes and REST Service Responses

Had an interesting IM exchange today regarding a REST service GET URI performing a database look up for user account information by email address and what HTTP response codes are appropriate for different response conditions.

The simple, successful "Here's the user info for that address" is easy-- HTTP 200 OK with the user info in a JSON payload. What about the other possibilities though?

The question prompting the conversation was "What happens when the email address isn't tied to any user? Is that a 404?" My assertion was no, a 404 means something else and there's a better existing HTTP response code.

This got me thinking about different scenarios and what HTTP response codes are appropriate for a simple GET request. Just as a red stoplight isn't an error, several different response conditions can and should result in different response codes with documented meanings.

Here's my list:

200 OK - Success, body payload contains current response payload
204 No Content - Success but nothing found matching your request, no body payload
301 Moved Permanently - Deprecated URI used, Correct current URI given in the Location response header, body payload contains response appropriate for the deprecated URI
400 Bad Request - Unsuccessful, bad email address format, no body payload
401 Unauthorized - Unsuccessful, incorrect/invalid authentication provided or missing, no body payload
404 Not Found - Unsuccessful, incorrect/invalid URI used, no body payload
410 Gone - Unsuccessful, unsupported old URI used, no body payload
500 Internal Server Error - Unsuccessful, server error, including web service processing issue and any database issues
503 Service Unavailable - Unsuccessful, server overloaded or under maintenance, no body payload

That's a lot of work for one GET call, just scratching the surface for versioned APIs. Even those don't cover gateways, proxies, or different HTTP verbs. Worst of all, they're are just my opinion.

My answer then to the original question is HTTP 204 No Content for a valid, successful email address lookup not tied to a user, i.e. the equivalent HTTP response code for a successful database query returning an empty result set.

What do you think? Are there others missing for a simple GET service URI? Do you use one or more of these response codes in a different way?

Java Key and Trust Stores

Twice in the last three months I've been addressed roughly the same issue, the dreaded "PKIX path build failed" or more specifically...


Exception in thread "xxx" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target


Digging around I found a few pointers to standard documentation and a few good answers on the usual suspects Q&A sites. No doubt this topic has been covered, but since this issue is still causing some lost work cycles better spent elsewhere, here's my treatment.

First, some background info about certificates. Certificates are created by a trusted certificate authority signing a certificate signing request for a named subject, e.g. a server hostname or a specific user. The certificate signing request registers ownership of a public key to the named subject. The public-private key pair are then used for encrypting a communication channel.

Signing a certificate signing request establishes a transitive trust relationship between the signing request's subject and a trusted certificate authority--if you trust one of the certificate authorities signing the certificate, you trust the certificate. Multiple intermediate certificate authorities' signatures may exist between the root certificate and the certificate's ultimate subject. Each signature is itself a certificate (and public key) forming a path from the root certificate authority through zero or more intermediate certificate authorities to the subject's signed certificate. (1)(2)

Let's look at an example certificate. While the commands below show Linux command line, most will work elsewhere without modification as long as the proper tools are installed and available on the executable search path.

Connecting to the Red Hat customer portal will get a secure connection.


$ echo | openssl s_client -showcerts -connect access.redhat.com:443

CONNECTED(00000003)
depth=3 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary Certification Authority
verify return:1
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify return:1
depth=0 C = US, ST = North Carolina, L = Raleigh, O = Red Hat  Inc., OU = Web Operations, CN = access.redhat.com
verify return:1

...


(If you skip the "echo |" at the beginning of the command, hit Ctrl-D to end the openssl s_client connection.)

Quite a bit scrolls by and we'll take a look at that in a minute. First though is the certificate signature path. We see two certificates from VeriSign, one from Symantec, and finally one from Red Hat for the access.redhat.com subject. This simply shows the first certificate signed the second, the second signed the third, and the third signed the fourth. If you trust one of the certificates in the signature path, you trust the whole certificate. Additional validation rules exist of course, e.g. this certificate matches the hostname to which we are connecting, etc., but we're concerned about our exception for now.

Hey, we've got something called a path!

One piece of our exception should make more sense now. It's complaining about "path build failed."

Is the certificate path what's failing?

Yes.

Why?

The JVM can find none of the path's certificates in its trust store. Only one certificate is required to be found but the JVM struck out for every single one.

How do we fix it?

At the risk of sounding like a consultant, that depends. However, the most likely cause is a certificate signed by an unknown, and therefore untrusted signing authority's certificate.

Certificate trust stores usually hold many globally trusted certificates from around the world. These usually show up as root or high-level intermediate certificates during certificate validation. The rule still holds though--if you trust a certificate along the path, you trust the certificate.

Creating your own certificate authority is entirely possible by creating a self-signed certificate. However, if no one trusts your self-signed certificate authority, they also won't trust any certificates you sign. The self-signed certificate must be added to trust stores to be useful. There are many places where this can be beneficial and cost-effective but can also be abused in numerous ways.

If the certificate path is comprised of certificates not in the globally trusted set, e.g. a self-signed corporate certificate, you'll need to create a trust store with this certificate and configure your JVM to use the new trust store.

Let's look at the access.redhat.com certificates' details. For this, we'll incorporate the Java keytool.


$ echo | openssl s_client -showcerts -connect access.redhat.com:443 2>/dev/null | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | keytool -printcert

Certificate[1]:
Owner: CN=access.redhat.com, OU=Web Operations, O="Red Hat  Inc.", L=Raleigh, ST=North Carolina, C=US
Issuer: CN=Symantec Class 3 Secure Server CA - G4, OU=Symantec Trust Network, O=Symantec Corporation, C=US
Serial number: 56dbb5f8bcdeacc6f3cfc2fa6d336ab9
Valid from: Mon Aug 22 20:00:00 EDT 2016 until: Wed Aug 23 19:59:59 EDT 2017
...
Certificate[2]:
Owner: CN=Symantec Class 3 Secure Server CA - G4, OU=Symantec Trust Network, O=Symantec Corporation, C=US
Issuer: CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
Serial number: 513fb9743870b73440418d30930699ff
Valid from: Wed Oct 30 20:00:00 EDT 2013 until: Mon Oct 30 19:59:59 EDT 2023
...
Certificate[3]:
Owner: CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
Issuer: OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
Serial number: 250ce8e030612e9f2b89f7054d7cf8fd
Valid from: Tue Nov 07 19:00:00 EST 2006 until: Sun Nov 07 18:59:59 EST 2021
...


There are several other lines omitted covering encryption details, valid certificate uses, and such. These lines are what we need though.

We can see these certificates should be in a normal trust store. VeriSign and Symantec have been managing certificates for a very long time.

But let's look at another certificate, something like we'd find on an internal corporate network.


$ echo | openssl s_client -connect rhighley.remote.csb:8443 -showcerts

CONNECTED(00000003)
depth=0 C = US, ST = Texas, L = Flower Mound, O = RHC NA Enterprise Integration Practice, OU = RHC NA EI Practice Blog Content Examples, CN = rhighley.remote.csb
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = Texas, L = Flower Mound, O = RHC NA Enterprise Integration Practice, OU = RHC NA EI Practice Blog Content Examples, CN = rhighley.remote.csb
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = Texas, L = Flower Mound, O = RHC NA Enterprise Integration Practice, OU = RHC NA EI Practice Blog Content Examples, CN = rhighley.remote.csb
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain

0 s:/C=US/ST=Texas/L=Flower Mound/O=RHC NA Enterprise Integration Practice/OU=RHC NA EI Practice Blog Content Examples/CN=rhighley.remote.csb
i:/C=US/ST=Texas/O=RHC NA Enterprise Integration Practice/OU=RHC NA EI Practice Certificate Authority/CN=RHC NA EI Practice Intermediate CA


Well, if OpenSSL can't verify the certificate with its standard trust store, a JVM will have a difficult time also.

We can see the subject certificate here is for a server called "rhighley.remote.csb" which matches the hostname to which we're connecting. The certificate is issued by this "RHC NA Enterprise Integration Practice" organization with their own intermediate certificate authority. Who are these guys? The better question though is should you trust a certificate issued by them.

Since this is just an example, let's say we trust them. After all, that's my team and they're good people!


$ echo | openssl s_client -connect rhighley.remote.csb:8443 -showcerts 2>/dev/null | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | keytool -printcert

Owner: CN=rhighley.remote.csb, OU=RHC NA EI Practice Blog Content Examples, O=RHC NA Enterprise Integration Practice, L=Flower Mound, ST=Texas, C=US
Issuer: CN=RHC NA EI Practice Intermediate CA, OU=RHC NA EI Practice Certificate Authority, O=RHC NA Enterprise Integration Practice, ST=Texas, C=US
Serial number: 1001
Valid from: Sun May 21 14:48:03 EDT 2017 until: Mon May 21 14:48:03 EDT 2018


This looks legit. Let's create a JKS file out of it. For that, we'll need to write out the intermediate certificate's Base64 content we have been piping to keytool's printcert subcommand then read the file with another keytool subcommand, importcert.


$ echo | openssl s_client -connect rhighley.remote.csb:8443 -showcerts 2>/dev/null | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >rhighley.remote.csb.pem

$ keytool -importcert -file rhighley.remote.csb.pem -keystore rhighley.remote.csb.jks

Enter keystore password:  <Enter a password>
Re-enter new password:  <Reenter the same password>
Owner: CN=rhighley.remote.csb, OU=RHC NA EI Practice Blog Content Examples, O=RHC NA Enterprise Integration Practice, L=Flower Mound, ST=Texas, C=US
Issuer: CN=RHC NA EI Practice Intermediate CA, OU=RHC NA EI Practice Certificate Authority, O=RHC NA Enterprise Integration Practice, ST=Texas, C=US
Serial number: 1001
Valid from: Sun May 21 14:48:03 EDT 2017 until: Mon May 21 14:48:03 EDT 2018
Certificate fingerprints:
MD5:  82:39:F8:C9:6F:A8:80:8D:C2:B2:5F:D5:64:92:C7:81
SHA1: AB:5D:23:7F:DA:BE:6A:5D:9E:15:A3:32:17:CC:C0:8A:72:C8:7D:19
SHA256: D5:1E:A0:9D:5D:75:BB:EC:9D:D6:09:64:DD:DC:38:BC:74:FA:82:0B:A4:A6:5A:68:CE:ED:3E:36:44:7A:0E:FF
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

... (All the certificate's extensions) ...

Trust this certificate? [no]: yes
Certificate was added to keystore


Now we have a JKS with the intermediate signing certificate ready for the JVM to use by specifying the javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword system properties. There are of course a number of ways to give these to the JVM. The command line is easiest.

java -Djavax.net.ssl.trustStore=rhighley.remote.csb.jks -Djavax.net.ssl.trustStorePassword=<password> ...

The trustStorePassword isn't strictly required since the keystore will only be referenced by the JVM, not modified.

One final word of caution. How the JVM treats trusted certificates has some sharp edges. By default, the JVM will look at its jre/lib/security/cacerts file for its trusted certificates. We are overriding this behavior through system properties. However, the JVM will consult one and only one trust store for certificates. Since we're providing a new trust store, it must contain ALL trusted certificates the JVM may encounter, including any globally trusted ones. Your new trust store cannot provide just a new trusted certificate or two and expect the JVM to "fall back" to the default certificate trust store containing the standard globally trusted set. (3)

Hopefully this helps explain what's going on with this "PKIX path build failed" exception and how to address it properly.

1. Wikipedia's certificate authority and associated articles are pretty good for a high-level understanding. https://en.wikipedia.org/wiki/Certificate_authority
2. Wikipedia's certificate path validation algorithm article: https://en.wikipedia.org/wiki/Certification_path_validation_algorithm
3. Oracle's Java Secure Socket Extension Reference Guide, Customizing JSSE section: http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization

Red Hat CDK DNS Issues

While not my usual wheelhouse, I've been struggling with a DNS issue with the Red Hat CDK on my home network. Not that I'm totally ignorant in the ways of configuring and running my own BIND daemons for fun and profit, this issue caught me off guard.

I run an internal BIND server on a Raspberry Pi running an Fedora 25 for ARMv7. While it resolved internal network IP addresses perfectly, I didn't have it configured as a caching DNS server for external addresses. Anything "outside the building" (my house) would go through my ISP's DNS servers because that's the list of DNS servers given to any machine on the network, whether by DHCP or manual configuration.

My CDK hosts' /etc/resolv.conf files looked like this:

# Generated by NetworkManager
search <My local domain name>
nameserver <My local BIND server's IP>
nameserver <My ISP router's IP>


Therein lay the failure.

Here's the behavior I saw. When starting the vagrant image, I'd see the following:

Bringing machine 'default' up with 'libvirt' provider...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
==> default: Registering box with vagrant-registration...
    default: Would you like to register the system now (default: yes)? [y|n]y
    default: username: XXXXXXXXXXX
    default: password:
==> default: Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.
==> default: Registering to: subscription.rhn.redhat.com:443/subscription


At that point, the CDK was running and was available for SSH. However, all DNS resolution attempts failed with "temporary failure in name resolution," including that required for Red Hat Subscription Manager registration and to pull new Docker images for the CDK.

Vagrant and libvirt use dnsmasq to provide a small DHCP service and DNS proxy to hosted containers. However, dnsmasq by default apparently only considers the first nameserver configured for the host. Since my host's first nameserver only resolved internal hostnames, redhat.com names were not resolvable.

The solution was nice and simple, thankfully. Just add a set of forwarders--in my case, my local ISP router and a couple of well-known DNS server farms.

forwarders {
  192.168.1.1;
  8.8.8.8;
  8.8.4.4;
};


Be careful with the BIND configuration options though. With recent DNS attacks causing all kinds of widespread havoc, there's plenty of opportunity for perpetuating someone else's bad behavior.

Inaugural Post

in•te•gral

(ĭn´tĭ·grəl, ĭn·tĕg´rəl)

• adj.
  
  1. Essential or necessary for completeness; constituent: The kitchen is an integral part of a house.
  2. Possessing everything essential; entire.
• n.
  
  1. A complete unit; a whole.

integral. (n.d.) American Heritage® Dictionary of the English Language, Fifth Edition. (2011). Retrieved October 19 2016 from http://www.thefreedictionary.com/integral

Introductions first. As you can probably tell from my profile, my name is Ryan. I'm currently working as an Architect in the Enterprise Integration Practice in NA Emerging Technologies Consulting at Red Hat. The Enterprise Integration Practice is focused on Red Hat JBoss Fuse and A-MQ, including the upstream Apache Camel, Karaf, ActiveMQ, CXF, and associated projects. A Practice Architect's role is split between client consulting engagements and knowledge sharing both within Red Hat and the worldwide developer community.

Intent next. Many avenues exist for knowledge sharing. Technical blogs are a common choice. My intention is to provide relevant content related to building enterprise-scale system integrations with open-source technologies and platforms, not just those with which Red Hat is involved.

In conclusion, a promise. The "in" alliteration ends now....