Category: Educational

Useful unix file editing commands

Find and replace text matching a regular expression in a single file

The above command searches for the (something\-[\da-zA-Z]+) regular expression and replaces it whole (because of the parenthesis which means to select the text matching the expression within) with ToReplace. The g in the end indicates that the operation will be applied to all matches in the file as supplied in path/to/file.txt argument. The -i parameter along with '' suggests the sed command to perform the edit on the file itself, without creating a new copy.

Find and replace text matching a regular expression in files matching name

The above command finds file within /base/directory whose names match *.txt format. Later we combine the output of this with xargs which appends each line of output (path to each matching file) to the following sed command. The -n 1 argument to the xargs command tells it to supply each line of argument one by one to the sed command.

Find and replace text matching a regular expression in files whose contents match a regular expression

In the above command we use the basic find command to get a list of files that we want to do the search in. Then we use grep to recursively find in those files. Now in the above command we don’t really need the -r flag for grep because the find command will list full paths to those files, but we would need it if we were doing find on a relative path instead (like .). The -l flag for grep here will only list the paths to the files that it found having the content TextToFindInFiles and not the actual matching contents like it usually prints. This list of the files is then outputted to the xargs command which then subsequently runs the sed command.

 

Posted on by Manthan Dave 0

Documenting project information with Maven

In most projects I’ve worked on, the project information is captured in some sort of README.md file. While this has its strengths, such as ability to write free form text and embed images, it doesn’t quite fit with the project because when the artifact is produced during the build time, the read me is not checked into the artifact repository along with it. So how do I know who the contributors were for that specific artifact version? How do I know where the project was hosted at the time?

Well, today I learned that maven provides ability to capture some of those attributes and some more! I like this because it means that when the artifact gets checked in (along with its pom.xml) the details about the project are captured for that version, frozen in time.

Great thing about these attibutes is that when you run mvn site they are used in the project information html page that maven produces. Technically, then you can upload this to any statically hosted site location, all versioned up and ready to be explored.

Ok so lets begin exploring these attributes. The first three attributes are the basic ones:

For people working in corporations and companies, the following might be useful, especially when open sourcing the project:

Additionally, the licenses tag can be used to describe some licensing information. How many times have you come across a project that has changed the licensing information half way through versions? This is a life saver (at least legally)!

Some source control information is also useful, however, whats more useful is knowing where to go in order to raise issues. In most companies, people use source control to store code but then use an alternative mechanism like Jira or Trello to mange issues. The scm and issueManagement tags are useful here in clarifying such information:

Developer information on projects is very useful. Companies don’t usually like this because, well, developers are supposed to be expendable, however, in my humble opinion, it is useful to list developers who were working on the project at the time. This way the versions can be tracked, at least to the lead working on it at the time. Use it thusly:

For contributors, people who have worked on the project, although not necessarily part of it exclusively, use:

Lastly, it is useful to capture some information about pipeline that was used to create the project. For this, maven has ciManagement tag:

The above is just a small example of what you can do with these attributes, there are many more sub-attributes that you can explore. Hope this is a good introduction and kicks off some ideas of what you can do with this information.

Posted on by Manthan Dave 1

How to delete all entries from Java JKS Keystore

I had to deal with this recently. After much trial and error, here’s the command that you can use to wipe your Java JKS Keystore of all its entries:

Here, the variable KEYSTOREis the path to your Java keystore and the variable KEYSTORE_PASS is the keystore’s password. If you are not comfortable in using the keystore password plain text in command line, I’d suggest you use an alternative version using a file containing keystore password or name of an environment variable instead. This will hide the password from appearing in shell history. You can do this by suffixing the -storepass argument with :file or :env resulting in it effectively becoming -storepass:file <path/to/file> or -storepass:env <ENV_NAME_WITHOUT_$. Here are some examples:

In the above, notice how the ${KEYSTORE_PASS} environment variable has changed to ${KEYSTORE_PASS_FILE}. Use this to provide a path to the file containing your keystore password.

Similar to previous, this one has been slightly modified to use the -storepass:env flag with ${KEYSTORE_PASS_ENV} environment variable instead.

Posted on by Manthan Dave 1

Making executable jar using maven

I was trying something out the other day and wanted to write a really simple application. So I created a simple application backed by Maven.

Now I could run the jar file that maven built using the standard -e flag that lets java know the entry point but all that is too main stream. I wanted maven to handle that for me.

After doing some googling, I found a plugin provided by codehaus. This one allowed me to run the application through maven. As you can see below, the configuration is quite simple.

Make sure you update the value inside mainClass with fully qualified name of your class that you want to run. This class must have a public static void main method in it.

Once you are happy with the configuration, you can run your application by executing the following in your terminal

While this is great, I cannot run the jar on its own on a server somewhere. If I wanted to, I’d have to get the source code with maven and then run it using the above command. Thats sub-optimal. So I went googling again.

Finally, I found this wonderful plugin provided by our friends at Apache. This is the maven-jar-plugin. This is a standard jar plugin but one of its features is ability to specify a mainClass attribute – just like the codehaus plugin. But unlike the codehaus plugin, I can run the jar as a standalone application without needing to pass in any other flags or parameters indicating the main class.

Here’s the maven build plugin configuration to use the maven-jar-plugin.

As it is standard with maven, package up your application using:

If the build was successful, just run your jar file using standard java -jar path/to/app.jar command. In my case, I ran:

 

Posted on by Manthan Dave 1

AWS Pipeline Plugin for Jenkins 2.x

I found this really cool plugin last week so I thought I’d make a post out of it.

I heavily use the “AssumeRole” capability within my application. Previously, this is what I used:

As you can see, the code above is using shell commands to achieve the assume role awesomeness. This works well when your jenkins job has a shell command step but not when you have a groovy pipeline defined in your Jenkins.

I struggled with it initially – one of the ideas I had was to upload the assume script somewhere like S3 and then pull it down and run it when I wanted to run commands under the assume-role. However, this felt a bit cumbersome.

Next thing in my mind was to write a groovy plugin that can do this for me. However, rather than reinventing the wheel, I started looking for existing solutions. Finally, I found the aws-pipeline-plugin.

Its a neat little plugin that allows you to do a bunch of basic stuff that you might want to do on AWS. Assume role is one of them.

So now with the new plugin, my code reduced to:

Here, I’ve got a couple of variables but their names should be self-explanatory of their purpose. The general idea is that anything you write within that withAWS block will get executed under the role specified in role variable.

Posted on by Manthan Dave 0