Thursday, January 31, 2019


Some basic facts about GCP VPC network:


 

A project:

Associates objects and services with billing.

Contains networks (quota max 5). This is not the limit and hence can be increased.

 

A VPC network:

Has no IP address range.

Is global and spans all available regions.

Contains subnetworks.

Can be of type default, auto mode, or custom mode.

An auto mode network can be converted to custom mode network, but “once custom, always custom.”

 

Subnetworks cross zones:

Subnetworks can extend across zones in the same region.

One VM and an alternate VM can be on the same subnet but in different zones.

A single firewall rule can apply to both VMs even though they are in different zones.

Defined by internal IP address prefix range

Specified in CIDR notation

○ IP ranges cannot overlap between subnets

○ IP range can be expanded but can never shrink

○ Specific to one region

Can cross zones within the region

Notice that the first address in the range, 10.0.0.1, is reserved for the "router" address. And the last address in the range, 10.0.0.255, is reserved for the "broadcast" address. VPC networks only support IPv4 unicast traffic. IPv4 broadcast and IPv4 multicast are not supported.

 

Networks have no IP range, so subnetworks don't need to fit into an address hierarchy. Instead, subnetworks can be used to group and manage resources. They can represent departments, business functions, or systems.

 

In the GCP VPC, that physical network structure doesn't exist. The network has no top-level IP range.

 

IP addresses:

 

  • Internal IP
    • Allocated from subnet range to VMs by DHCP
    • DHCP lease is renewed every 24 hours Reserved (static)
    • VM name + IP is registered with network-scoped DNS

  

  • External IP
    • Assigned from pool (ephemeral)
    •  VM doesn't know external IP; it is mapped to the internal IP
    • Reserved (static)

Billed when not attached to a running VM



       DNS resolution for internal addresses

      Each instance has a hostname that can be resolved to an internal IP address:

The hostname is the same as the instance name.

FQDN is [hostname].c.[project-id].internal.

○ Example: guestbook-test.c.guestbook-151617.internal

Name resolution is handled by internal DNS resolver:

Provided as part of Compute Engine (169.254.169.254).

Configured for use on instance via DHCP.

Provides answer for internal and external addresses.

 

            DNS resolution for external addresses

Instances with external IP addresses can allow connections from hosts outside of the project.

○ Users connect directly using external IP address.

○ Admins can also publish public DNS records pointing to the instance.

○ Public DNS records are not published automatically.

DNS records for external addresses can be published using existing DNS servers (outside of GCP).

DNS zones can be hosted using Cloud DNS.

○ Create zone and configure domain DNS to use.

○ Create, update, and remove records manually or via API

 

Using IP aliasing, you can configure multiple IP addresses, representing containers or applications hosted in a VM, without having to define a separate network interface.

Draw the alias IP range from the local subnet's primary or secondary CIDR ranges. Configuring alias IP ranges describes commands for setting up a subnet with

secondary ranges and for assigning alias IP addresses to VMs.

 

Routes:

Every network has:

Routes that let instances in a network send traffic directly to each other.

A default route that directs packets to destinations that are outside the network.

 

The fact that a packet has a route to a destination doesn’t mean it can get there; firewall rules must also allow the packet.

 

Firewall rules:

        Firewall rules protect your VM instances from unapproved connections

         Every VPC network also functions as a distributed firewall.

Firewall rules are applied to the network as a whole.

Connections are allowed or denied at the instance level.

Sunday, January 27, 2019

How to copy files from GCP cloud bucket to local machine using gsutil


How to copy files from GCP cloud bucket to local machine using  gsutil


If you are trying to download files from a GCP bucket storage to your local machine using cloud shell, you may encounter an error like:

CommandException: Destination URL must name a directory, bucket, or bucket

Running gsutil command in cloud shell(ephemeral VM) will not of much help here and it works fine if we intend to copy from one storage bucket to another bucket.

However, our requirement is to copy the file to the local system. The steps will be:

  1. Download GoogleCloudSDKInstaller.exe and install it on your local system
  2. Then open Google cloud SDK and it will prompt you to connect to your GCP account
  3. Then it will ask the project name in which the intended file to be copied is present
  4. Command to pass:
     
            gsutil cp <source> <destination>

            For example:

                      gsutil cp gs://punamtestbucket/test.txt  F:\punam\GCP_1


Some other basic stuff around GCP bucket:

  • It give option to upload files and folder from local machine to bucket however download option is not available and for that we need to use the gsutil commands

                    

  • Create a bucket

            Buckets are the basic containers that hold your data in Cloud Storage

             gsutil mb -l us-east1 gs://punamtestbucket2


             If bucket name already exists then will get an exception as:
       
                          ServiceException: 409 Bucket my-awesome-bucket already exists.
                          Try again with a different bucket name.


  • List contents of a bucket or folder

            gsutil ls gs://punamtestbucket2

  
  • Make your object publicly accessible

  1. Use the gsutil acl ch command to grant all users read permission for the object stored in your bucket:

    gsutil acl ch -u AllUsers:R gs://punamtestbucket/test.txt
          If successful, the command returns:

                Updated ACL on gs://punamtestbucket/test.txt
          Now anyone can get your object.
  1. To remove this permission, use the command:

    gsutil acl ch -d AllUsers gs://punamtestbucket/test.txt

  • Give someone access to your bucket

  1. Use the gsutil iam ch command to give a specific email address permission to read and write objects in your bucket:

    gsutil iam ch user:jane@gmail.com:objectCreator,objectViewer gs://punamtestbucket/test.txt
          Now someone else can put things into and view what's in your bucket.
  1. To remove this permission:

    gsutil iam ch -d user:jane@gmail.com:objectCreator,objectViewer gs://punamtestbucket/test.txt


Point to be noted is:
Use ACL(Access Control List) when need to give permission at object level whereas use IAM(Identity Access Management) when bucket level permission needs to be provided

  • Delete objects
 
                      gsutil rm gs://punamtestbucket/test.txt

  • To delete bucket and its content

            gsutil rm -r  gs://punamtestbucket