alanwilliamson

A simple Java API class for Amazon SimpleDB

We recently had cause to utilise the latest addition to Amazon's Web Services platform, SimpleDB. There are a number of Java libraries to get up and running with, and each one has their strengths and weaknesses.

When I coded up the SimpleDB for inclusion into OpenBD, I used the typica library, which is a great set of API's. Although recently, for another project, I just wanted something real quick, and while typica is nice, it does have some funky JAR file dependencies that makes it non trivial to install at times.

I found this chap, which is a single class that does everything inside one class and does not require any external JAR files that are not within the JDK.

Not a single XML parser in sight. It merely packages up the REST call to Amazon, doing all the authenication token passing, and then dumps you the resulting response back. Nice and simple.

And it works beautifully!

I decided to add to it though, with a very crude parsing algorithm (no SAX/DOM parser) to pull out the results from the XML packet and put them into something the calling function can use. Amazon's XML is so simple you don't need the extra overhead of those fully rounded parsers in 99% of the time.

So instead of calling listDomains(), which returns the raw string, you can now call listDomainsList() that will return a java.util.List of domains. I've added the same for the other methods where you need the actual data back out from the web service call.

Thanks to Ian Schumacher for developing the original class and making it available on Amazon under a very liberal license (Absolutely free in all senses of the word, absolutely no restrictions, no attribution necessary).

In the same spirit, I include the modified version of that code here, to be used in the same spirit as the original author. Just Use It!

import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.TimeZone;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

import sun.misc.BASE64Encoder;

public class SimpleDB extends Object {

 public SimpleDB(final String id, final String key) {
  this.id = id;
  secret = new SecretKeySpec(key.getBytes(), "HmacSHA1");
 }

 static Mac mac;
 static {
  try {
   mac = Mac.getInstance("HmacSHA1");
  } catch (final Exception e) {
   e.printStackTrace();
  }
 }

 private byte[] hmacSha1(final String str) {
  try {
   mac.init(secret);
  } catch (final Exception e) {
   throw new RuntimeException(e);
  }
  return mac.doFinal(str.getBytes());
 }

 static String getString(final InputStream is, final String charEncoding) {
  try {
   final ByteArrayOutputStream out = new ByteArrayOutputStream();
   final byte ba[] = new byte[8192];
   int read = is.read(ba);
   while (read > -1) {
    out.write(ba, 0, read);
    read = is.read(ba);
   }
   return out.toString(charEncoding);
  } catch (final Exception e) {
   e.printStackTrace();
   return null;
  } finally {
   try {
    if (is != null) {
     is.close();
    }
   } catch (final Exception e) {
    e.printStackTrace();
   }
  }
 }

 public static String httpDate() {
  final String dateFormat = "yyyy-MM-dd\'T\'HH:mm:ss\'Z\'";
  final SimpleDateFormat format = new SimpleDateFormat(dateFormat, Locale.US);
  format.setTimeZone(TimeZone.getTimeZone("GMT"));
  return format.format(new Date());
 }

 private final String id;

 private final SecretKeySpec secret;

 String base64(final byte b[]) {
  return new BASE64Encoder().encode(b);
 }

 public String createDomain(final String name) throws Exception {
  final String time = httpDate();
  final String toSign = "ActionCreateDomainAWSAccessKeyId" + id + "DomainName" + name + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=CreateDomain" + "&AWSAccessKeyId=" + id + "&DomainName=" + name + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }

 public String deleteAttributes(final String domain, final String item, final Map<String, String> map) throws Exception {
  final String time = httpDate();
  final StringBuffer signPart = new StringBuffer();
  final StringBuffer queryPart = new StringBuffer();
  int count = 0;
  for (final Map.Entry<String, String> x : map.entrySet()) {
   signPart.append("Attribute.").append(count).append(".Name").append(x.getKey());
   signPart.append("Attribute.").append(count).append(".Value").append(x.getValue());
   queryPart.append("&Attribute.").append(count).append(".Name").append("=").append(URLEncoder.encode(x.getKey(), "utf-8"));
   queryPart.append("&Attribute.").append(count).append(".Value").append("=").append(URLEncoder.encode(x.getValue(), "utf-8"));
   ++count;
  }
  final String toSign = "ActionDeleteAttributes" + signPart + "AWSAccessKeyId" + id + "DomainName" + domain + "ItemName" + item + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=DeleteAttributes" + queryPart + "&AWSAccessKeyId=" + id + "&DomainName=" + domain + "&ItemName=" + item + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }

 
 public String deleteDomain(final String name) throws Exception {
  final String time = httpDate();
  final String toSign = "ActionDeleteDomainAWSAccessKeyId" + id + "DomainName" + name + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=DeleteDomain" + "&AWSAccessKeyId=" + id + "&DomainName=" + name + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }

 
 public String getAttributes(final String domain, final String item) throws Exception {
  final String time = httpDate();
  final String toSign = "ActionGetAttributes" + "AWSAccessKeyId" + id + "DomainName" + domain + "ItemName" + item + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=GetAttributes" + "&AWSAccessKeyId=" + id + "&DomainName=" + domain + "&ItemName=" + item + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }
 
 
 public HashMap getAttributesAsMap(final String domain, final String item) throws Exception {
  String result = getAttributes(domain,item);
  
  HashMap map = new HashMap();
  List attributes = getElements( result, "Attribute" );
  for ( int x=0; x < attributes.size(); x++ ){
   String t = attributes.get(x).toString();
   
   String key = t.substring( t.indexOf("<Name>") + 6, t.indexOf("</Name>") );
   String val = t.substring( t.indexOf("<Value>") + 7, t.indexOf("</Value>") );
   map.put( key, val );
  }
  
  return map;
 }

 public String listDomains() throws Exception {
  final String time = httpDate();
  final String toSign = "ActionListDomainsAWSAccessKeyId" + id + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=ListDomains" + "&AWSAccessKeyId=" + id + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }
 
 public List listDomainsList() throws Exception {
  return getElements( listDomains(), "DomainName" );
 }
 
 public String putAttributes(final String domain, final String item, final Map<String, String> map) throws Exception {
  final String time = httpDate();
  final StringBuffer signPart = new StringBuffer();
  final StringBuffer queryPart = new StringBuffer();
  int count = 0;
  for (final Map.Entry<String, String> x : map.entrySet()) {
   signPart.append("Attribute.").append(count).append(".Name").append(x.getKey());
   signPart.append("Attribute.").append(count).append(".Value").append(x.getValue());
   queryPart.append("&Attribute.").append(count).append(".Name").append("=").append(URLEncoder.encode(x.getKey(), "utf-8"));
   queryPart.append("&Attribute.").append(count).append(".Value").append("=").append(URLEncoder.encode(x.getValue(), "utf-8"));
   ++count;
  }
  final String toSign = "ActionPutAttributes" + signPart + "AWSAccessKeyId" + id + "DomainName" + domain + "ItemName" + item + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=PutAttributes" + queryPart + "&AWSAccessKeyId=" + id + "&DomainName=" + domain + "&ItemName=" + item + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }

 
 public String query(final String domain, final String q, final String tok) throws Exception {
  final String time = httpDate();
  String tokstr = "", utokstr = "";
  if (tok != null) {
   tokstr = "NextToken" + tok;
   utokstr = "&NextToken=" + tok;
  }
  final String toSign = "ActionQuery" + "AWSAccessKeyId" + id + "DomainName" + domain + "MaxNumberOfItems250" + tokstr + "QueryExpression" + q + "SignatureVersion1Timestamp" + time + "Version2007-11-07";
  final String endpoint = "https://sdb.amazonaws.com/" + "?Action=Query" + "&AWSAccessKeyId=" + id + "&DomainName=" + domain + "&MaxNumberOfItems=250" + utokstr + "&QueryExpression=" + URLEncoder.encode(q, "utf-8") + "&SignatureVersion=1" + "&Timestamp=" + time + "&Version=2007-11-07" + "&Signature=" + URLEncoder.encode(base64(hmacSha1(toSign)), "utf-8");
  final URL url = new URL(endpoint);
  final HttpURLConnection con = (HttpURLConnection) url.openConnection();
  if (con.getResponseCode() >= 300) {
   throw new Exception(con.getResponseMessage());
  }
  return getString(con.getInputStream(), "utf-8");
 }
 
 public List queryAsList(final String domain, final String q, final String tok) throws Exception {
  return getElements( query(domain,q,tok), "ItemName" );
 }

 
 private List getElements( String text, String elem ){
  ArrayList list = new ArrayList();
  
  String sTag = "<" + elem + ">";
  String eTag = "</" + elem + ">";
  
  int c1 = text.indexOf( sTag );
  int c2 = text.indexOf( eTag );
  while ( c1 != -1 && c2 != -1 ){
   list.add( text.substring( c1 + sTag.length(), c2 ) );
   c1 = text.indexOf( sTag, c2 );
   if ( c1 != -1 )
    c2 = text.indexOf( eTag, c1 );
  }
  return list;
 }
}

Adapated from the original code submission.

Comments

Thank you for the kind words and I love your enhancement of the code.

left by Ian Schumacher — Thursday, 17 July 2008 2:07 PM

Have you tried calling it a number of times. It will be limited by the size of the URL in a single GET. So you may be hitting against that.

Remember also, a single element can only be 1K, thats an Amazon limitation, not a Java class one.

left by Alan Williamson — Friday, 4 July 2008 10:14 AM — web site

HI ,

I have question regarding the simpledb upload program, in Java. The program is itself gr8 and runs gr8. Is ther anything wrong with putAttribute function, i mean r this its limitations.

But when i try to write data with more than 10 attr to a single domain, it fails. It works fine with ruby but i need it in java. It also works fine when attributes are less then 10. Can you tell am I going wrong. some where or its the limitation of the method we are using.

Karan Cal State University, Fullerton

left by Karan — Thursday, 3 July 2008 9:19 PM

Leave Comment

please note, all comments will be moderated for spam and abuse before being publicly posted.

Article Details