Dive into Delightful Message Queues with AWS SQS

 

 The heart of the AWS SQS show? It’s the messages, bustling with data like little digital bees! Each message finds a cozy home in a queue, waiting patiently to be processed by a single, dedicated subscriber.

Amazon SQS throws on its invisibility cloak for a little while to prevent other consumers from grabbing it too soon. This cloak, called a visibility timeout, ensures each message gets its moment to shine and be processed properly. By default, this cloak lasts for 30 seconds, but you can adjust it to fit your needs — from a lightning-fast 0 seconds to a leisurely 12 hours.

Once the subscriber finishes the job, the message bids farewell and departs the queue, leaving space for the next one in line.

But wait, there’s more to this queueing magic! SQS offers a nifty feature called long polling. Imagine a queue full of eager messages, but instead of constantly checking if there are new ones (which can be tiring!), Amazon SQS sends a response after it collects at least one available message, up to the maximum number of messages specified in the Receive Message request. Amazon SQS sends an empty response only if the polling wait time expires.

Speaking of smooth, let’s talk about retries. Sometimes, even the most dedicated subscribers might encounter hiccups. That’s where the maxReceiveCount comes in. You can use this setting to tell SQS how many times it should attempt to process a message before moving on. This limits the number of times Lambda will retry to process a failed execution. As SQS natively supports Dead Letter Queue DLQ, it’s possible to create it and configure SQS to transfer failed message to that Queue.

But messages aren’t all about the content, they can carry extra information too! Think of it like a little luggage tag with details like timestamps, location data, or even a digital signature. These message attributes travel alongside the message itself (separately from the body), providing valuable context for the recipient.

Delay queues add another layer of fun to the mix. When a message enters the SQS, it takes a pre-defined break before being consumed. This is perfect for situations where tasks need to happen at a specific time. It’s possible to set delay seconds on individual messages, rather than on an entire queue. To do so, use Message timers to allow Amazon SQS to use the message timer’s DelaySeconds value instead of the delay queue’s DelaySeconds value.

With these delightful features at your fingertips, AWS SQS can help you build resilient, efficient, and scalable applications. So, what are you waiting for? Start queuing up for some cloud-based magic!

Join the conversation and share your SQS experiences in the comments below!

Good To know:

• Message size restriction is 256KB
• The default waiting time of a message in SQS Queue is 4 days
• For Lambda function consuming from SQS, the best visibility timeout is 6x the function timeout. This gives time for retries with extra buffer.
• To support cross-regions or SQS active mode, it’s possible to use SNS that delivers messages to SQS Queue in different regions. This is useful to divide customers by location for access to the nearest region.

Understand the VPC endpoint like a Pro and get ready for the AWS certification

Imagine sending your data on a private jet ✈️ instead of a bus 🚌 through rush hour traffic 🚦. That’s the magic of VPC endpoint: a secure, high-speed lane connecting your AWS resource directly to specific services, all within the safe confines of the AS network.

This blog post is your-first class ticket to understanding these powerful tools and get ready for you AWS certification. Buckle up and we will explore the two types of VPC endpoints:

• Interface endpoints: Think of them as private tunnels, shipping data directly to the service without ever touching the public internet.
• Gateway endpoints: these acts as secure gateways, allowing communication with AWS S3 and DynamoDB.

 As explained before, a VPC endpoints lets you privately connect your VPC to supported AWS services and VPC endpoint services.

VPC endpoints are virtual devices. They are horizontally scaled, redundant, and highly available Amazon VPC components that allow communication between instances in an Amazon VPC and services without imposing availability risks or bandwidth constraints on network traffic.

Gateway endpoints

A gateway endpoint is a virtual connection within your VPC that allow resources in your VPC to directly access AWS S3 and DynamoDB. The gateway targets specific IP routes in an Amazon VPC route table, in the form of a prefix-list, used for traffic destined to Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3).

When you create a gateway endpoint, you select the VPC route tables for the subnets you enable. A route with a prefix-list is automatically added to each selected route table. Those routes can’t be modified or deleted unless you disassociate the route table from the gateway or you delete the gateway itself.

There is no additional charges for using the gateway endpoint, which makes it preferable over interface endpoint in case your need access to S3 or DynamoDB. Nevertheless, Gateway endpoint doesn’t allow access from on-premise networks, from peered VPC in other regions and through transit gateway. For those use cases, you must use the Interface endpoint.

Here short tips when working with the gateway endpoints:

• A gateway endpoint is available only in the region you created it. Be sure it’s the same region as your S3 bucket.
• You can have routes to S3 and/or DynamoDB in the same route table. But you can’t have multiple routes to the same service in a single route table.
• The instances access S3 or DynamoDB service using its public endpoints. Their security groups must allow traffic to and from the services using the prefix_list_id as destination with the TCP protocol on port 443.
• The Network ACLs for the subnets for theses instances must also allow traffic to and from the services.

You need to consider the AWS specificities in traffic routing. AWs uses the most specific route that matches the traffic to determine how to route it (longest prefix match). In route tables with endpoint route this means:

• If the route that sends all internet traffic (0.0.0.0/0) to an internet gateway, the endpoint route takes precedence for traffic sent to S3 or DynamoDB in the current region.
• Traffic destined for S3 or DynamoDB in a different region goes to the internet gateway as prefix lists are specific to a Region.
• If there is a route that specifies the exact IP address range for the S3 or DynamoDB in the same region, that route takes precedence over the endpoint route.

Interface endpoints

Interface endpoints enable connectivity to services over AWS PrivateLink.

But wait, what is PrivateLink. It’s an amazing service that allows you to to securely connect your resources in an Amazon Virtual Private Cloud VPC to specific AWS resources, other VPCs, and even on-premise applications, without ever exposing your data in the public internet.

Here some benefit of the PrivateLink:

• AWS PrivateLink gives on-premises networks private access to AWS services through Direct Connect.
• You can make services available to other accounts and VPCs that are accessed securely as private endpoints.
• If you use AWS PrivateLink with a Network Load Balancer to route traffic to your service or application, clients can connect to any service you host.
• Services configured to support AWS PrivateLink can be offered as a subscription service through the AWS Marketplace.

The interface endpoint is powered by the AWS PrivateLink. When configuring an interface VPC endpoint, an elastic network interface (ENI) with a private IP address is deployed in your subnet. This ENI acts as an entry point for traffic destined to a supported service. Remember, attaching a Security Group is crucial for access control.

Provisioning an interface endpoint incurs hourly charges based on its uptime and data processing.

To optimize costs and simplify management, consider hosting interface endpoint in a centralized VPC. All the “spoke” VPCs can then leverage these centralized endpoints via Transit Gateway or VPC peering, eliminating the need for individual endpoints in each VPC. This approach known as Centralized VPC endpoint architecture pattern, minimizes costs and streamlines access control with centralized Security Groups.

P.S. Don’t forget to share your thoughts and experience with VPC endpoints in the comment below!

Unlocking the Power of AWS Virtual Private Cloud VPC: Essential Notes for your AWS Certification

Ever felt overwhelmed by the vast amount of information you need to study for the AWS certification exam, specifically when it comes to VPC? I’ve been there, and like many others, struggled to grasp the intricacies of this vital service. Fortunately, while preparing for my exam, I found some key notes that significantly improved my understanding.

In this post, I’m excited to share these notes with you, offering a clear and concise overview of VPC. We’ll dive into its core components, explore connectivity options, and wrap up with essential “good to know” tips. Whether you’re a complete beginner or someone seeking a refresher, this post aims to demystify VPC and prepare you for your AWS certification journey.

Well, let’s begin with the definition of VPC. A virtual private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch AWS resources, such as Amazon EC2 instances, databases, into your VPC.

VPC is a regional service. To start, AWS provides a default VPC that comes with public subnet in each availability zone, default Network ACL, default Security Group, Main route table, an internet gateway already configured and settings to enable DNS resolution.

The non-default VPC that you creates is an isolated network that do not allow anything in or out without explicit configuration.

As shown in the AWS VPC console, Subnets are how you add structure and functionality to your Amazon VPCs, and they are an Availability Zone resilient feature of AWS.

A subnet is a sub-network of your Amazon VPC CIDR range that is created in one Availability Zone. The are a range of IP addresses in your VPC. You can launch AWS resources, such as EC2 instances, into your subnets. Each subnet resides entirely within one Availability Zone.

When assigning CIDR to the Subnet, the first 4 IP addresses and the last IP address are reserved to the VPC and used for Network Address, VPC router, mapping to amazon-provided DNS, for future use, the last is for broadcast.

You can create the following resources for your VPC: custom Network ACL, route table, SG, Internet Gateway, NAT GW

Components in a VPC

• CIDR: CIDR is based on the idea of subnet masks. A mask is placed over an IP address and creates a sub network. The subnet mask signals to the router which part of the IP address is assigned to the hosts and which determines the network
• Subnet that runs in One Availability Zone AZ. In case of AZ failure, the subnets and all running services inside it will fail. For high availability, it’s recommended to pace services in different subnets from different AZs.
• EC2 are virtual servers that provide scalable computing capacity while eliminating the need to invest on hardware. EC2 instance is launched in a private or public subnet within the VPC. The instance receives a primary private IP address from the IPv4 address configured for the subnet.
• A VPC Router is highly available and moves traffic from A to B. The router runs in all your Availability zones that your Amazon VPC uses. The router has a network interface in each subnet and uses the first reserved IP address of the subnet. the router is fully managed by AWS to route traffic between subnets in your VPC.
• Route table that contains a set of rules called routes that determines where network traffic from subnet or gateway is directed. They are created at the VPC level, but are associated with a subnet. If you don’t associate a new route table to your subnet, then the main route table is associated. Only one route table can be associated with a subnet, but it’s possible to use the same route table for many subnets at a time.
• Internet Gateway sits on the edge of your VPC and allow traffic to and from the Internet. A VPC can have only One Internet Gateway IG, and it’s regional. It works across all Availability zones. Once created, the IG should be attached to the VPC and added to the route table for the subnet that becomes public.
• Network Access Control List NACL is a type of security filter that filter traffic that enters and leave a subnet. A NACL is attached to a subnet. Hence, it only manages traffic that is crossing the subnet boundary but not internal traffic. Network ACLs are stateless, which means that if you add a rule to allow/deny an inbound traffic, then you must also add the same rule for outbound traffic.
• Security Groups are another security feature that complement the NACL. It handles security outside the subnet but also the interaction inside the subnet. Security groups are attached to the elastic network interface of AWS resources in the subnet. It sits at the boundary of EC2 instances. Security groups are stateful and view traffic as one stream. If traffic is allowed in, then that traffic is automatically allowed back out.

Connectivity

The different connectivity option for the VPC are:

• The internet via an internet Gateway (One per VPC)
• Your corporate data center using Site-to-site VPN connection (using Virtual Private Gateway)
• Other AWs services (via internet gateway, VPC endpoints, virtual private gateway)
• Other VPCs in the same or different region as well as same or different account via VPC peering or transit gateway

Good to know:

• CIDR (Classless Inter-Domain Routing) allocated to VPC are the private one (Class A: 10.0.0.0, Class B: 172.16.x.x, Class C: 192.168.x.x).
• Subnets can’t span across multiple AZs
• Max CIDR per VPC is 5.
• CIDR should not overlap within your other networks
• For resiliency, it is recommended to provision subnets in at least 2 AZs.
• You cannot increase or decrease the size of an existing CIDR block that’s associated with the VPC.
• You assign a single CIDR IP Address range as the primary CIDR block for the VPC and can add up to 4 secondary CIDR. Subnets within a VPC are addressed from these CIDR ranges by you.
• It’s also possible to shrink the network by removing secondary CIDRs.
• While you can create multiple VPCs with overlapping IP address ranges, doing so will prohibit you from connecting these VPCs (example via VPC peering). For this reason it’s recommended using non-overlapping IP address ranges.
• CIDR IP address range for VPC can be between /28 (in CIDR notation) and /16 in size.
• The minimum size of a subnet is a /28 (or 14 IP addresses.) for IPv4. Subnets cannot be larger than the VPC in which they are created.
• Amazon reserves the first four (4) IP addresses and the last one (1) IP address of every subnet for IP networking purposes.

Fix Python packages imports not found

 I have started working with Python and Apache-Airflow in a data engineering project. But one I run the code, I had this error:

In the  same time, the packages airflow presents an issue and seems not found. hmmm.

Checking the Apache-airflow under python\site-packages, the dependencies are available.

Checking further, I found that Visual studio code is using the version 3.7.7, while my terminal is pointing to a newer version (3.12).This happened when I have updated Python, and added it to the environment variables, but not on Visual Studio.

So what happens in that when I install the packages, it's available in the newer version, but not found in Visual studio as the older Python version doesn't contain the requested package.

In order to fix that, on visual studio, get the Python interpreter:

ctrl + shift + p then type: Python: select interpreter

 

 

 Then select the Python version your environment variable is pointing.

That's all 😉.

Fix AWS Elastic BeansTalk Roles Error

AWS Elastic Beanstalk is a service provided by Amazon Cloud in order to deploy and manage web application in a simple way. This post focus on an error that you can get when you try to create for the first time a beanstalk application and environment (dev, staging, production.

When working with Elastic Beanstalk, it's possible to have this error "aws-elasticbeanstalk-ec2-role doesn't exists when creating the environment". This is due to the fact that Amazon is no more creating the role and it should be manually added.

When creating an environment to run the application on EC2 instances, 2 roles are required. 

aws-elasticbeanstalk-service-role:

- AWSElasticBeanstalkManagedUpdatesCustomerRolePolicy 

- AWSElasticBeanstalkEnhancedHealth

aws-elasticbeanstalk-ec2-role:

- AWSElasticBeanstalkWebTier 

- AWSElasticBeanstalkWorkerTier

- AWSElasticBeanstalkMulticontainerDocker

so in order to fix the errors "No Environment found for EnvironmentName" and "aws-elasticbeanstalk-ec2-role doesn't exists when creating the environment", you should add the new rol in the IAM roles. To do so:

    - Open IAM console

    - Choose Roles and create a new Role

    - Choose "AWS Service" as  a trusted entity

    - For the use Case, use EC2

    - Attach AWSElasticBeanstalkWebTier, AWSElasticBeanstalkWorkerTier, and AWSElasticBeanstalkMulticontainerDocker.

    - Last, don't forget to create the Environment using the already created role for EC2.

How to succeed your Interview at Amazon

Assalamou Alaykom Everybody, and welcome back to Me. Yes I am actually back to the Dev and I will be sharing with you some tips as usual. Hope you keep always your positivity as you need it everywhere.

So, you may want to find a job at Amazon, which I think is an amazing company to work in. Just by reading their leadership pillars, you can understand easily how wonderful to work with such people.

It was my case. So i will share with you some tips to know before looking for a job at Amazon or google.

My first advice is to get prepared for the interviews before even applying. In fact, I have learnt just to get the required time to get ready before starting to search for new job. It may be even 1 year before begining. Oh yes that was surprising for me but I understood that to get really ready, you may require all this time.  

So the first move is to well prepare your CV, it should be attractive and concise. Well this is so hard to achieve especially for me, but I got a good advice from a recruiter who told me : "Short resume gives us the curiousity to know more about you". That day, I absolutely understand that I need to learn how to commercialize my resume and introduce some curiousity in the mind of the recruiter, who finally don't know me and receives handred of application daily. 

Think about that and add some positivity and special things in your Resume in order to make it more attractive. As i like design, I have decided to make a little bit of colors, a touch of joy as I don't like black resume. And trust me it works because you get out from the fleet with a special signature. So try to add your own signature that shows something of your personality and may. 

Now that the resume is ready, you need to prepare yourself to tell a story about your old jobs. Not a boring story, but a good ones:

•  What was your contribution and the result with tangible data if possible
• The challenges and Mistakes or failures
• What you would do better 
• Did you get any conflicts
• How you manage deadlines and problems

In fact, it's true that during the interviews we talk just superficially. We don't try to show exactly what we have faced during our projects, to talk frankly about some bad experiences. But I have learnt that we should come back to those experiences and describe what was successful and a distaster. 

At the end don't forget that we are presenting ourself as a human being not just a bench of technologies. I advice you also to have a look on Emotional Intelligence which is very important as part of your culture and personality. In fact you may have an extarordinary technical background, but the fact that you lack some intelligence in deeling with people may cause disasters. To be more successful, you need to learn how to share knowledges, encourage people around you and motivate them. So don't forget to show that also.

Now coming back to some tips with Amazon interview. The first one is a technical and small behavior test. And trust me, it is not easy to succeed it even if you are a senior software engineer. Yes, indeed, those tests require some knowlege about algorithms that we don't use in our daily tasks.

You may say what't the need of such tests. It was my first thought, but when I had a look on those tests, it was just amazing how lovely and challenging they are. I may stay nights without sleeping thinking about a tricky solution. 

So, as you are looking for Amazon position and preparing for an interview, try first to get ready with those nice problems and solve many of them before applying. One of the best website to get ready is leetcode.com. It's just gorgious not only because you write your solution but also you can compare it with other solutions in terms of performances and that's an amazing thing.

You can create an account on  leetcode.com and start with easy, medium then hard problems. It will give you a good chance to integrate Amazon or Google by getting trained and explaining your solution, performances and used strategy.

Another resource that you can use is a very good book named "Cracking the Coding Interview 189 Programming Questions and Solutions". It gives good details and strategies about how to get ready for interviews with big companies like Amazon, Apple, and Google. You can fin it online and take somtime reading it but it's worthy.

That's my advices for today. Hope for you the best and share with me your tips, i would like to hear from you. 

Enjoy the Day.