Deploy GoLambda With MongoDB and AWS SAM (Part 3)

Micro-service trials and tribulations

SAM-MongoDB-Go[pher]

Do you like not having a reference to build something? As well known as these three technologies are, I was hard-pressed to find another article demonstrating a model of them working in tandem. In fact, I didn't find one. So, I gave birth to this one!

I wanted to start the New Year with a new language. I also wanted to document some of the things that I learned along the way. This article is more about explaining what I did to get Go running locally and my experience developing locally with SAM, Go, and the Mongo driver. I will take you on my journey from setup and then walk you through the nine or so methods that make this API. Ahora, let's begin!

This article assumes that you have some experience with Lambda functions and SAM, and MongoDB but none using them with Go. If you don't, you must first have Golang, SAM, the AWS CLI, and Docker. This was a weekend project for me and my introduction to Go. Please enjoy!

After you have everything installed you're ready to work.

First, you need to cd to your Go directory. I use a Mac, so for me the directory was originally located in $HOME/go. It's important where you put this since all of your Goroutines will need to live here in thesrc/ directory — more on that in a moment¹. So, since I like to personalize things, I decided to move it to my projects/ directory.

¹This has changed since the addition of Modules.

In order to do that and still have things work, I needed to update my$PATH. Since I use Zsh, I needed to update my .zshrc with the following lines:

export GOPATH=$HOME/porjects/go
export PATH="$PATH:$HOME/projects/go/bin"

Run the following to make the dir for your Goroutine:

$ mkdir $HOME/projects/go/src/test-go

All of my Go programs will need to live here $HOME/projects/go/src. This is known as the Go workspace. You can read more about Go Workspaces here.

Then I run the following to create a boiler-plate Hello World Golang Lambda using the SAM CLI:

$ cd $HOME/projects/go/src/test-go
$ sam init — runtime go1.x — name test-go

When you run the above command in your terminal, you get this.

.
└── test-go
    ├── Makefile
    ├── README.md
    ├── hello-world
    │   ├── test-go
    │   ├── hello-world
    │   ├── main.go
    │   └── main_test.go
    └── template.yaml

Next, we want to run:

$ make deps
$ make build
$ sam local start-api

From a new terminal window, you can run curl localhost:3000/hello or simply open it in your browser.

If all of your aws-cli credentials are in place, the first time you run $ sam local start-api youll see something like this in your console:

Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2019-12-15 13:23:18  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking hello-world (go1.x)
2019-12-15 13:23:35 Found credentials in shared credentials file: ~/.aws/credentials
Fetching lambci/lambda:go1.x Docker container image....................

Afterward, you'll be able to invoke your lamba either using curl http://localhost:3000/hello or via the browser by visiting the same URL as I mentioned above.

Great! You're up and running and you're developing, but now's a good time to talk about a couple of gotchas.

• Now all of that was assuming that you didn't already initiate test-go/ as a Git repository. If you did and try to run the $ make deps command you may run into an error similar to the following:

$ make deps
go get -u ./...
# cd /Users/jahagitonga/projects/go/src/test-go; git submodule update --init --recursive
fatal: No url found for submodule path 'test-go' in .gitmodules
package aahs-go-back-end/aahs-func/test-go: exit status 128

• SAM will not hot-reload Golang lambdas natively, which is terrible. In light of that, we need a Node package named supervisor to help us watch for changes. Yup, we need Node, so if you don't have it, go here to find out more about it and installing it on your machine.

Now let's update our Makefile. My Makefile looks like this:

The idea for this came from Ucchishta Sivagur. His repo can be viewed here. Thanks uccmen! If you update your Makefile run $ make && make watch, you'll be ready to develop with hot-reloading enabled for your Golang Lambda. Sweet!

• When building a lambda, you will see this error locally if you forget to declare the method in your template.yaml file:

{
   "message": "Missing Authentication Token"
}

So make sure you update your template.yaml file accordingly and don't be alarmed although the message is highly misleading. Hope the AWS crew can get a better error message out there soon!

Getting MongoDb Up and Running!

Since this is a small project, I'm not concerned with replication or sharding; I just needed a MongoDB instance in the cloud, so I went with mLab, since it is free and they're a company owned by MongoDb. If I need to upgrade and harness the power of replication and sharding in the future, I can migrate to Atlas with no problems.

We'll need a few dependencies to make this work:

"github.com/joho/godotenv"

"go.mongodb.org/mongo-driver/bson/primitive"

"go.mongodb.org/mongo-driver/mongo"

"go.mongodb.org/mongo-driver/mongo/options"

"go.mongodb.org/mongo-driver/mongo/readpref"

bson "go.mongodb.org/mongo-driver/bson"

Build the Lambda

We can add the above to our import statement of our main.go file. A complete list of all the packages that I used can be found here in the repo. Let's start looking at func main():

I like to keep things simple, so the first thing func main() is going to do is get our environment variables, check to see if there is already a Mongo client and that mongoURI isn't assigned a value of an empty string. Once we do that we connect to our MongoDb instance, or start the lambda.

One of the first things that I noticed coming from JavaScript is that Go needs a little help when dealing with environment variables. You can make them and access them at a lower level using your system and the os package that Go gives you. But if you're accustomed to using .env files for storing these variables rather than on your system you'll need to import “github.com/joho/godotenv”. Even in Node you need the dotenv package so that the Node process will gather these env vars from your system and your .env file(s). And to access them all you need isproces.env.YOUR_VAR_NAME. But we're not dealing with the simplicity of Node! Check out the getMongoURIEnvVar method:

This function just takes a name of type string. First, we have to load the .env file then we can use the os package to retrieve the value assigned to that name. Notice how the Load method only returns an error — interesting assumption. If there wasn't an error it must have worked, right?

Well now that we have all of that, let's take a look at how we connect to our mongo instance:

Here you can see that I'm first checking to see if the client is not nil. If it isn't the program will continue and use the cached instance that is available for connection pooling (best practice). If it's nil, we create a connectionError variable since the short-hand declaration/assignment := will not interfere with us using client globally on line 7. We also create a context in which our function will run and connect to the db (we have some basic logging for error handling in there too). In high-level terms this context says that if the program hasn't connected in 15 seconds to stop and move on. Go has great documentation on how Google uses context.

If we pay attention to and pattern our usage of context after their usage, we can develop super-efficient APIs as well. I do a decent job here but would love for the more seasoned gophers to give me some pointers on how things can be optimized here — pun intended! You want to pass context around to ensure efficiency in your app and that there's proper release of those resources once things have been executed within their context. Context gets deep, so please check out the two resources I listed in this paragraph for the lo' on all the things of context.

After we've connected, or not, the next step in our Goroutine is to start our handler:

I try to keep it as simple as possible (KISS), right? Notice that I create another context. This is the context in which our CRUD operations run. I will pass this down to my functions and once they have returned this Goroutine will run the defer call to cancel() which will cancel the context, thus freeing up those resources. Next, I'm going to check that the client is nil. If it is, which it shouldn't be, the program will send an error to the client alerting the end-user that a connection couldn't be established — in so many words.

I use a simple switch statement to decide what happens next. I chose to do this because the project is small and I am only using onecatchall route to handle my request. If the project were bigger, I probably wouldn't, but seeing how this is an example of using Go, MongoDb and SAM, just know that everything here isn't completely up to the RESTful spec — but we're really close!

With that being said, let's look at func getStories().

First things first, I ping the client. I do this to make sure things are on the up and up. If they aren't — the program throws an error. But let's say that things are, then we execute a find on our collection, assess it for errors and use the cursor's .Next()method to loop over the cursor and decode its elements into something that our Goroutine can use later on. We then append them to a stories list and return our call to func marshalJSONAndSend() passing it our list. The last thing done after that return executes is the deferred call to cursor.Close(ctx).

But what about func marshalJSONAndSend(). Well, let's take a peek:

This little guy is going to take our list of stories, make it into a data type — []byte — that can then be converted to a string, line 7, and sent to the client via our API Gateway Response.

That's that!

I've been using this func handleError() but haven't shown it. There's not much to it. Take a look for yourself.

We're almost done. We have the ability to read from the db but let's create a document. We do that with our func postStory().

I follow the same pattern that's used in the other methods, so no change there but I did need to do some type conversion so that it could properly be added to my db. Once I've done that, I used a variable, newDbResult, to create a map of the results according to their type. I did this to dry up some code that was used to handle the result and send the appropriate response.

Before we jump to func handleResultSendResponse() let's first take a look at the Story struct. Structs remind me of interfaces in TypeScript (remember I come from JavaScript land). They are used to enumerate the fields of a piece of data. They contain names as fields and the value type associated with said fields. They also contain metadata that is of critical importance to the proper functioning of our API. Check out the Story struct:

Without that metadata to the right of our field types, our struct would not map to our JSON/BSON schema. We also needed to capitalize the field names. See what this guy had to say about capitalizing field names of structs.

https://github.com/asaskevich/govalidator/issues/187

Now what about func handleResultSendResponse()?

I commented the code so, there you have it. The important thing here is reflection. Golang uses reflect to allow us to use Go in a more dynamic fashion, i.e., examine types at runtime versus compile time. Reflection deserves its own post, so much that I found a good one here, in addition to the Go docs here and their blog here. Once we've properly reflected, we're able to fmt.Sprintf a stringso that we could send it back to our client, line 18 of func handleResultSendResponse(). Knowing reflection allowed me to DRY up some code for reuse in other places. You gotta love that!

The last thing we have is the update function. Let's dive in!

First, let me say that I had to get used to all of the type conversion that is required to make this API run. If someone has a better way or optimizations please feel free to drop me a line or two in the comments.

First we declare a variable. Take the update that we have and convert it to []bytes. On line four I love the way Go allows us to do some shorthand assignment and the use of those variables in our if/elsestatements. Those are scoped to the if block of course, and its respective else should it have one. In fact, I liked it so much that I was willing to go against Go's linter to use this syntax with an elseblock that did nothing but return a function call. I ended up refactoring the code.

When we look at our filter and upDate vars we see that the Go driver uses BSON. This was new to me and the double curlies made me think of Angular. Here's a great tutorial on using the Go driver.

It was now time to deploy. To deploy follow the instructions in theREADME.md that SAM provides. It's one of the simplest ways to, declaratively speaking, to get you in the cloud next to CDK.

Although I must say, when I did go live, I still had a bug to work out. Originally I had placed a few variables in the global scope. One of which was being used in func getStories(); it was the stories []Story list. Locally this was working fine but in the cloud where things were optimized, each time I made a get request the lambda would return one more of the same story — I only had one in the Db.

It took me about 15 minutes to realize what was the potential problem. The best I could come up with was that the context of the lambda was outliving the actual request and each time it received a new request it appended a new document to the collection. The best course of action was to use the stories collection in the scope of func getStories(). Once I did that things were back to normal.

That's all I have. I hope this benefits someone and shaves off a couple of hours of research. Let me know how I can improve, what you would have done differently, and what you liked!

Thanks for reading!