How to create a software stack for Ambari

How to create a software stack for Ambari
March 24, 2015 Abdiel Aviles

NOTE: It’s been a while since I wrote this blog or worked with these technologies. This post has been very popular and I thank all for this. That being said, I won’t be able to continue answering questions and thus will close all comments. Thanks again to all who have shown interest.

In this post I will try to give a more detailed explanation on how to build a customized Ambari Stack. I’ve decided to share all that I’ve learned through reading Ambari Server’s source code. This is my way of giving back to the community.

What is Ambari? and What is an Ambari Stack?

If you are reading this it is because you already know what Ambari is and what Ambari Stacks are. But still let’s refresh our memories.

What is Ambari, from the Ambari website:

“The Apache Ambari project is aimed at making Hadoop management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs.”

In simpler words, Ambari is a Resource Management Platform. That is, you have your cluster and it lets you install and monitor distributed apps (for Hadoop), easily.

As of this writing, Ambari provides an explanation to what and how to build custom Ambari Stacks. If that link was good enough for you, great! you are done. But it was not as clear for me and that’s why I took the time to go through their code and existing stacks. After many many hours I ended up developing an Ambari Stack for Spark 1.2. Preparing a stack for Spark is not as straightforward as it is the case for other services, mostly because of it’s topology (if you have specific questions regarding a Spark Stack shoot me an email). But that is not what I’m going to show you here, instead I’m going to give you a template source code that you can use with your apps. That being said, keep in mind that this article is written from a “hacker” perspective.

Oh yes, what is a Stack? What I understand is that an Ambari Stack is akin to an RPM or a DEB. It is simply a service installation package for distributed clusters managed by Ambari. I chosed the word “service” since that is how Ambari refers to the apps it manages. HDFS is installed from a stack, HBASE is installed from a stack, Zookeeper is installed from a stack, Storm is …. well you get the idea.

Since version 1.7.0 they allow you to install custom Stacks or services from the Ambari Web UI. This is not the same as Ambari Views and I might spare a blog later for this.

NOTE: Download the source code from the link found at the end of this post. OPEN all files and refer to them as we go through the sections.

Folder and File Structure

This is how the (simplified) folder structure, of an Ambari Stack definition, looks like.

The most important file here is metainfo.xml and it defines what the stack is, how to install it, what commands are available, and what are the handles (the scripts that do the work). Let’s wee what’s inside it.

Custom Commands

The customCommand tag works the same as the commandScript tag but it differs in that it must contain a method definition with the same name as the custom command specified by the tag. Custom commands are then injected into the Ambari web interface and can be executed from there. Go back to the Command Script section and read through. There should be another subroutine defined (along with start, stop, etc.) with the name of the custom command you are describing here.

Required Services

The requiredServices block tells Ambari which services must be installed prior to this stack/service. So in simpler terms, this are stack dependencies of stacks. For example, if your service needs HDFS and YARN to be pre-installed, then you would have something like:

In our stack we are requiring the presence of the HDFS stack and the YARN stack. already installed in the cluster.

Configuration Dependencies

Configuration dependencies is one of the most useful features of the Ambari Stack. It allows the user to modify the service/app configuration through the Ambari Web UI. These configurations could be used from any of the subroutines defined in the command scripts. These values are defined inside an xml file. Recall from the metainfo.xml file shown before, there is a configuration dependency called my-config-env. This is telling ambari to look for a file named /config/my-config-env.xml. Inside this file we can define as many variables as we need. All of them will be available for re-definition through the Ambari Web UI. Let’s look at my-config-env.xml:

We have just defined a couple of variables: dummy_user with a value of dummy, and dummy_var with a value of 123. We can now use these variables from inside our scripts, and also modify them from the Ambari Web UI. For example, let’s take the install subroutine inside the command script defined before. We need to create a new user for our Dummy Service. We could do this as follows:

In the script above we are loading configuration variables (line 5), installing dependencies (line 11), creating a new group (line 14) and a new user (line 15). Note that we are using a few convenience methods provided by the resource_management package: Script.get_config(), Execute() and format(). The command Script.get_config() will load all configuration the variables that have been defined inside configuration files, and from all services. This means that you can access variables defined in other services as well as your own variables as seen in line 7, where we bind the dummy_user configuration with a local variable.

Important Global Variables

There are other very important global variables that are useful to us. For example, during installation we might need to know who are the master and slaves nodes. This is defined during the installation wizard in the Ambari Web UI and obviously cannot be known in advance. Thankfully we can grab these values with the following snippet:

The masterNode variable now contains the FQDN name of the master node configured during installation. Similarly the slaveList variable contains an array with the FQDN names of the slaves nodes. It is also important to note that these variables were found from these two variables: dummy_app_master_hosts and dummy_app_slave_hosts. As you can see these variables are named after the name you assigned your service in metainfo.xml: DUMMY_APP.

There are other interesting variables you can find inside the config object. For example you can find the Ganglia host through config['clusterHostInfo']['ganglia_server_host'][0], and in the same manner every other configuration of all installed services.

The role of the start() handle in configuring your service

The start() handle, as defined in the Command Scripts section, must always check if configurations new configurations are in place. As a rule of thumb, always re-configure your service files when starting the service. This will ensure that your service will be up-to date when a configuration change has been made from the Ambari Web UI.s

Installing your brand new Stack

Copy Files

In order to install your stack, simply copy all your files to the following path in the Ambari Server host:

/var/lib/ambari-server/resources/stacks/HDP/2.2/services/DUMMY_SERVICE

Afterwards you must restart your ambari server with the following command:

Your service structure must follow the one described in the “Folder and File Structure” section.

Installation Wizard

  1. Login to Ambari Web UI,
    Ambari Login

  2. From the Ambari dashboard,
    Ambari Dashboard

  3. Select the Actions>Add Service.
    Add Service

  4. The Installation Wizard will be shown. Select the service to be installed,
    Wizard Select Service

  5. Select the master node,
    Wizard Select Master

  6. Select the slave nodes,
    Wizard Select Slaves

  7. Configure the service,
    Wizard Configure

  8. Review the installation settings,
    Wizard Review

  9. Monitor the installation,
    Wizard Installing

  10. Once the installation is complete, click Next to exit the wizard.
    Wizard Installation Complete

  11. Now your dashboard shows the newly added service.
    Ambari Dashboard with Service

Service Actions

Each service has an administration panel where you can monitor and control the service. Click on the service name.
Dummy Service

If you need to change any configuration, go into the Configs panel. Here you can edit the current service configurations or roll back into previous versions. Every time you edit the configurations, the service must be restarted in order to apply the changes. Recall from the “Configuration Dependencies” section, that the start() handle will be in charge of editing the actual configuration files with these new settings.

There are various service level actions you can take through the UI.
Dummy Service Actions

Also, if you go to directly into the host through the Hosts tab, you could perform component level actions. For example you could stop just the master component of your service.
Dummy Service Component Actions

Once an action has been performed, Ambari will automatically adjust the UI to show the current status. For example if you do a service level Stop, the UI should look like this.
Dummy Service Stop

To restart the service simply select the service level Start command.

Wrap Up

We have seen how to describe your new service in the metainfo.xml. We have also seen how to install and control your service through the handles inside the command scripts. We saw how to allow for further configuration through the configuration xml files. And finally we went through the installation and control of our Dummy Service.

As any good hacker you might already be building your own distributed app in your head. I hope this was helpful and feel free to drop a line if you can’t figure out how to do something else. There were many many other details that were not covered in this blog. Our aim is to present a simple but yet useful example that would bootstrap your Ambari Stack development.

About Us

At Mozart Analytics LLC, we ask your data how to make you more money. We specialize in collecting and analyzing huge amounts of data from many sources, to provide real-time insights to our customers as a tool for aiding them in making informed and accurate decisions.

Source Code

Fork the code from https://github.com/mozart-analytics/ambari-stack-template

Hacker / Engineer / Data Junkie, Abdiel is a Co-Founder of Mozart Analytics and our lead Data Guy.

54 Comments

  1. baris acar 2 years ago

    Hello Abdiel,
    Do you have a working stack for Spark? I have just started trying to build spark 1.4.1 stack on ambari, and checking the bundled spark. I would be grateful if you could share your version or any insights.

    Thanks
    Baris

    • Author
      Abdiel Aviles 2 years ago

      The one I developed was for Spark 1.2.0 in Ambari 1.7, not Spark 1.4.1. If you are looking to use Streaming then it will not work for you. What version of Ambari are you developing for?

    • Author
      Abdiel Aviles 2 years ago

      BTW, the latest supported version by Ambari 2 is Spark 1.3.1

  2. baris acar 2 years ago

    I am trying both on 2.0.1(spark 1.2) and 2.1.0 (spark 1.3) branch.
    Also trying to initiate a docker dev env.
    I guess I will have to build spark deb files.

    • Author
      Abdiel Aviles 2 years ago

      If you manage to run Ambari within dockers let me know. I tried and failed due to my lack of knowledge with Docker networking and links. Checkout the guys at SequenceIQ. They seem to have built this.

  3. Amey 2 years ago

    Hi
    I don’t want to hardcode my service version into metainfo.xml, Can I do it?

    DUMMY_APP
    My Dummy APP
    This is a distributed app.
    0.1 ————————————– Here I want a place holder, can I do it?

    • Author
      Abdiel Aviles 2 years ago

      The short answer is no, you cannot. The longer answer is that metainfo.xml is used at installation time of your service. To install your service, ambari-server copies all the files into each of the nodes and metainfo.xml is static as well as everything else. The only place where you can put variables (placeholders) is in the configs. If you are upgrading your service, then you will need to deploy another stack and then Ambari will allow you to do the upgrades.

  4. Amey 2 years ago

    Thanks Abdiel,
    I have more questions:
    1. How can I do hosts specific configuration using apache Ambari? Can I do it or whatever configuration I have selected applies for all the hosts?
    2. Is there any Ambari way that I can check my service is installed and running properly?

    • Amey 2 years ago

      Hi Abdiel,

      I have one more question,
      In my custom service’s “service-site.xml”, I have a property as below:

      properties.file
      Content of –properties.file.

      k1=v1
      k2=v2

      but I want that this propety value to be picked up as an actual file from that location

      properties.file
      /opt/myProduct/config/myProduct.properties

      By giving this new property ambari displays properties.file value as above mentioned file location, but instead I want to pick up that actual file. How can I do that?

      Regards,
      Amey

      • Author
        Abdiel Aviles 2 years ago

        Regarding “service-site.xml”, I’m going to assume this is a configuration file.
        I don’t think that is possible, to load a file into a configuration variable/property. Since I don’t know exactly what you need, the closest thing I would think of is to use a template. Let me give you an example. Again, this might not suit your purposes but it gives you another capability.

        With a template you will go the other way around, you will be generating your properties file from within your python scripts.

        <!–THIS IS INSIDE A CONFIG XML FILE–>
        <property>
        <name>myProduct_properties_template</name>
        <value>k1={{my_v1}}
        k2={{my_v2}}
        </value>
        <description>myProduct properties file template.</description>
        </property>

        Take a look at the following snippet: k1={{my_v1}}
        The my_v1 is a variable that you can define inside your python code, and with those values then generate your properties file. The way you do this is as follows:

        #set your variables
        import params
        params.my_v1="v1"
        params.my_v2="v2"
        env.set_params(params)
        File("/opt/myProduct/config/myProduct.properties",
        owner=params.myProduct_user,
        content=InlineTemplate(params.myProduct_properties_template)
        )

        Here we are using the method InLineTemplate() to let python replace your properties inside the template. Also, we are building a File with the resulting text.

        I must warn you, I have not tested this and it might not work at all (I hope is does).

        Let me know if it does 🙂

    • Author
      Abdiel Aviles 2 years ago

      Great questions actually.

      1. How can I do hosts specific configuration using apache Ambari? Can I do it or whatever configuration I have selected applies for all the hosts?
      This is one I never thought about and since the last time I used Ambari (1.7) it was not possible directly from Ambari for specific nodes. BUT, depending on what you want to achieve, you might have a shot at it.
      Let me explain. Looking back at the Configuration Dependencies section, you can see that we have a single configuration file for all the hosts (masters and slaves). But also remember a couple of useful details. First, 1- each host (master or slave) will have a start() method defined. And every time you change a configuration in Ambari, a re-start is scheduled for all nodes. So, everytime you change a config, you will execute the start() at each and every host. So from there we go into the second useful detail, 2- you will always have access to all configuration variables from the start() method. This is explained at the Configuration Dependencies section. You even have host information as explained at the end of that same section.
      So, in other words, if you are able to deduce your configurations from inside the start() method, then you can do it since you will be running that method at each of the nodes with full access to all the global configurations.

      2. Is there any Ambari way that I can check my service is installed and running properly?
      Definately. There are two things you can do, a service check, and a status check. In a service check you will run something against your service and make sure it returns fine. For example, in Spark what I do is to run the SparkPi example. The second and most critical is the status check. If you do not implement this correctly then Ambari will not work and will either always show your nodes as running or as dead.

      I’m going to give you the status check I used for my Spark service implementation. But, I strongly recommend that you get a look at how the Ambari guys do it on each of the services. The idea behind the status check is to try and find the PID of the service and check if it is alive. But if you have a custom service (not a third party service) then it will be your responsibility to register a file containing the PID in a public folder when the service starts. This file is the one your service check will be looking for. Here’s the code:

      # this is a snippet from the master.py file
      def status(self, env):
      import glob
      import params
      env.set_params(params)
      print 'Status check for the Apache Spark Master' #This will be printed on the logs that are shown in Ambari
      check_process_status(glob.glob(params.spark_master_pid_file)[0])
      pass

      See how easy the guys at Ambari made it for us? The method check_process_status() was provided by them. The only thing we had to do was give it the location of the PID file. So, if your service already generates a PID file with a known location, the just feed that method with it and Ambari will take care of the rest.

      Let me know if you still have questions.

      • Amey 2 years ago

        Thanks Abdiel for your elaborate answers,
        1. How can I do hosts specific configuration using apache Ambari web UI is what I meant?
        When I select a particular host, and click onto particular service, that service configuration is for all the hosts that run that service.
        2. Is there any Ambari way that I can check my service is installed and running properly?
        Yes I too have status method defined and passed a PID file to it, but how can I verify that it is running properly with Ambari web UI?

  5. Shishir 2 years ago

    Hi Abdiel,
    I have a question regarding ambari stack :
    How to use theme framework with ambari to provide theme like combo,slider to the property? I need to add a toggle with yes or no property for licence.
    I am following the below link but not able to see themes for the customize service.
    http://search-hadoop.com/m/g7Bc62I7Ncfx9tB1&subj=Re+Using+theme+framework+on+my+own+stack
    Steps which i have done:
    1. Added theme.json entry in metainfo.xml
    2. Created theme.json as:
    {
    “configuration”: {
    “placement”: {
    “configuration-layout”: “default”,
    “configs”: [
    {
    “config”: “general/licence”,
    “subsection-name”: “subsection-ganeral-col1”
    }
    ]
    },
    “widgets”: [
    {
    “config”: “general/licence””,
    “widget”: {
    “type”: “toggle”
    }
    }
    ]
    }
    }
    3. In general.xml present in configuration folder as:
    Added following entry:

    licence
    for licence
    Licence

    true
    Enabled

    false
    Disabled

    1

    But still it is not reflecting it.

    Regards,
    Shishir

    • Shishir 2 years ago

      Point 3 is xml file(general.xml) which is not reflected properly.
      So written in different format to give you an idea:
      <property
      licence</name
      for licence</description
      Licence</dispaly-name
      <value-attributes
      <entries
      <entry
      true</value
      Enabled</label
      </entry
      <entry
      false</value
      Disabled</label
      </entry
      </entries
      1</selection-cardinality
      </value-attributes
      </property

    • Author
      Abdiel Aviles 2 years ago

      Shishir,

      Templating is a new feature on Ambari 2.1+ and was not available when I wrote this. I really have no experience on this but it sure looks like a very useful feature. So, for now I don’t have an answer for you but I’ll try to find better documentation for this.

  6. Shishir 2 years ago

    <property
    <name licence</name
    <description for licence</description
    <dispaly-name Licence</dispaly-name
    <value-attributes
    <entries
    <entry
    <value true</value
    <label Enabled</label
    </entry
    <entry
    <value false</value
    <label Disabled</label
    </entry
    </entries
    <selection-cardinality 1</selection-cardinality
    </value-attributes
    </property

  7. Shishir 2 years ago

    Thanks Abdiel Aviles.
    Am able to use theme framework correctly and now am able see different wigets like combo,slider in ambari.
    But i have one small question from you that how can we format text inside property?
    EG:
    I have property like:
    (using different language as xml doesnot show up here)
    Property-open
    name-open
    licence
    name-close
    display-name-open
    Apache licence agreement
    and complete licence
    display-name-closed
    value-open
    true
    value-close
    value-attributes-open
    entries-open
    entry-open
    value-open
    true
    value-close
    label-open Accept label-close
    entry-close
    entry-open
    value-open false value-close
    label-open Don’t Accept label-close
    entry-close
    entries-close
    value-attributes-close
    property-close

    i need to format content inside “display-name-open” and “display-name-close” where i thought to put complete licence. But between display-name tag it is not taking \n or tab.
    Any idea how to format the text in property?
    or suppose i need to give 2-3 tabs or \n in name of property then how to do it?

  8. Jorge 2 years ago

    Hello Abdiel,
    I am trying to install spark 1.2.0 using ambari 2.1.0 using sless11 and virtual box, and I try to override the
    /var/lib/ambari-server/resources/stacks/HDP/2.3/metainfo.xml file with the one that is located in
    /var/lib/ambari-server/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml
    but when I select the components to install, it still shows on the UI the version 1.3.1 (after an ambari-server restart)
    Does ambari uses a default version to control this?, is there a way to instal a downgrade version of any of the components?
    Many Thanks!

    • Author
      Abdiel Aviles 2 years ago

      Hi Jorge,

      I’m not sure that’ll be the way to go for a couple of reasons. A few issues so far, 1- Even if you specify a different Spark version in the metainfo.xml, you will need the correct repo files installed, otherwise your distro will not find the repo to install Spark 1.2. But the good news is that you can modify the ones that you have already installed on your distro. Open them with a text editor and edit the Spark repo urls. 2- If you don’t see any changes on the UI, then you are either not changing the correct files or not restarting the service. Before fixing this let me explain you something. The first thing that should go up is the ambari-server, once started it will check for any registered nodes online and distribute the stack (files, scripts, configs). It will do this through the ambari-agent. Now back at you not seeing the changes in the UI. First try restarting both the ambari-agent and the ambari-server. If that does not work, you might need to check if there is a folder called cache in either the server or the agent installation folders (I think it’s in the agent). You must make sure that both the server and all your agents get your changes. Inside the cache folder look for the stack and make the appropriate changes. Restart all.

      EDIT: I also forgot to mention that you must make sure that your version of HDP is compatible with Spark 1.2… but I’m sure you already checked that. Also, cross your fingers as the Spark stack you are editing contains various scripts targeted at Spark 1.3.1. I’m almost sure this will break.

      • Jorge 2 years ago

        Thank you very much for your comment, greatly appreciated, in fact the version that I have to install is the 1.3.1, but I just spawned a server with a different version for conveniency and assumed that the process would be the same. Still very useful answer!

  9. Amey 2 years ago

    Hi Abdiel,

    What is the procedure for providing kerberos support for custom service.
    What are the steps that are required for custom service to be kerbersised?

    Regards,
    Amey

    • Author
      Abdiel Aviles 2 years ago

      Sorry, I’ve not worked with that.

  10. Amey 2 years ago

    Hi Abdiel,

    Anything for Security (Kerberos) with custom service ?

  11. Prasad 2 years ago

    Hi Abdiel,

    Is there any way by which I can directly debug my python scripts on the machine hosting the ambari server instead of going through the UI.

    • Author
      Abdiel Aviles 2 years ago

      You know… it is painful. I’m not a python guy so can’t really give you much of an insight. When I built this I was in a time crunch and had to settle for logs (seen from the Ambari logs) and manually deploying the scripts to each of the nodes. The only hint that I can give you is that there is a cache on the ambari-agent where you should put your updates.

      BTW: Sorry for the late reply. I’ve been overloaded for the past few months.

  12. Priyanka 2 years ago

    Hi,
    I have a question about what you mentioned in OS Specifics section.
    Case 1: add your repository to the hosts:
    Can you point to some example, where do we need to add our repository?
    Case2: If you do not have a packaged app, you might still be able to do your own installation through the stack handle scripts:
    Can you please provide some details on how to do this.
    Case3: You have a package and if you want to push it to HDP repository is it possible?

    • Author
      Abdiel Aviles 2 years ago

      Hi Priyanka,
      1- The osSpecifics section allows you to install dependencies. Internally Ambari will run the “yum” or the “apt-get” command to install these packages. So let’s suppose you have an RPM for a dependency called “my-dep” that has not been published anywhere, Ambari will run something like “yum install my-pkg” but this will fail. You will have to update (manually) the repositories of the host and give it the address of where to find “my-pkg”.
      2- If you do not have a packaged dependency then you are free to perform the installation through the python scripts in your stack. That is, you will have to execute code to make the installation from inside the python scripts of your stack. This could be anything, so it’s hard for me to give you a concrete but generic example. All I’m doing is suggesting alternatives to using repos and packages.
      3- No you cannot publish to HDP repository. You will have to host your own repo but this is not that difficult either.

  13. Gitan 2 years ago

    Could you please explain why the following works:
    def status(self, env):
    import params
    print ‘Checking status of pid file’
    dummy_master_pid_file = “/var/run/Application/Application.pid”
    check_process_status(dummy_master_pid_file)

    But when I try to pass in a parameter to check_process_status, Ambari believes that the process dies off after a minute or so — even though I still see everything running fine on the server.

    def status(self, env):
    import params
    print ‘Checking status of pid file’
    check=format(“{params.pid}/Application.pid”)
    check_process_status(check)

    • Author
      Abdiel Aviles 2 years ago

      Try this:

      def status(self, env):
      import params
      env.set_params(params)
      print ‘Checking status of pid file’
      check=format(“{params.pid}/Application.pid”)
      check_process_status(check)

      • Gitan 2 years ago

        I gave this a shot and still havn’t had any luck. Ambari just sets the process as stopped after about a minute, with nothing thrown to any logs that I can see. However when I go to the server, I can still see the service actively running and writing things to the logs.

        • Author
          Abdiel Aviles 2 years ago

          I would need a bit more context to give you a better assessment. Can you share more of your scripts? I would like to see your imports and your config file. It would be better if you share a github gist. Also, which Ambari version?

          In the meantime, try this:

          def status(self, env):
          import glob
          import params
          env.set_params(params)
          print ‘Checking status of pid file’
          check=format(“{params.pid}/Application.pid”)
          check_process_status(glob.glob(check)[0])

          • Gitan 2 years ago

            Here’s a gist with the status section from master.py, params.py and custom-config.xml
            https://gist.github.com/ze-gitan/7c941eef1906d37f0310

            Let me know if this makes any more sense. I tried the clob suggestion and Istill have the same issue.

          • Author
            Abdiel Aviles 2 years ago

            Change

            check=format("{params.pid_file}/Custom.pid")

            for

            check=format("{pid_file}/Custom.pid")

            And if that is still not working then get the path with glob.glob.

            check=format("{pid_file}/Custom.pid")
            check_process_status(glob.glob(check)[0])

  14. Ankit 2 years ago

    I was wondering what does the method .execute() do and how does it work? Example in DummyMaster().execute()I am an experienced java/scala programmer, but have very little knowledge in Python. Google didn’t help this time.
    Also, how to I get the PID for the service after starting? I tried looking up the documentation for Execute() and ExecuteHadoop() on the below link, but the link says *** DRAFT – Being Edited ***
    https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Python+Libraries

    • Author
      Abdiel Aviles 2 years ago

      Welcome to the club of “Java People Hacking Python with Poor Documentation”. That was the main reason for me doing this post. Anyways, to answer bluntly your execute() question, I don’t know. But my best guess (and seen behaviour) is that execute is injected by Ambari. The Ambari Services Stack is an opinionated framework in order to be able to plugin new functionality. There are expected methods and a fixed structure to follow. That’s all explained in the post. But I’m sure I missing stuff. At the time of writing the docs were not too helpful, not sure about their state now. To answer your second question about the PID, you must generate a PID file when your service starts. If you are wrapping a service like I did with Spark, then the service might already generate the PID and you’ll have to figure out where it is. After you either generate or find your PID, then you can define the status check as I explained above in the “Component Status” section. And finally regarding the Execute() and ExecuteHadoop() methods, I don’t think there’s documentation about them. But, if you are like me and enjoy (not really) reading someone else’s code :), they are defined (as of version 1.7) inside the ../resource_management/core folder. I only briefly explained them above, but the code is there. Enjoy!

      • Ankit 2 years ago

        Wow, thanks for the help! After a good three days, I have managed to integrate my demo application which does nothing more than logging the current date into a text file 🙁 Now since you are the best help I have, I can’t resist being a little greedy and ask one more – This is an extension to Amey’s question above. I need to integrate an application which has it’s app-specific parameters available in some properties (eg. – k1=v1) files and some XML files. Now I partly understand from the above answers that is no Ambari-way of listing these property-parameters to the UI, and I probably would need to chalk out some naive, ungraceful way to achieve the same (Please correct me if I am wrong on this one). Also, is it possible even to read XML files which are located outside of the default configuration directory( in which ‘my-config-env.xml’ is located in the example)? Is there a way we can specify the external path of our configuration directory?

        • Author
          Abdiel Aviles 2 years ago

          Three days? That’s great! I took me 30.
          So let me see if I understand you correctly. You have a few properties you want to configure through Ambari dashboard. That’s perfectly possible, look at the “Configuration Dependencies” section. With that option you can control whatever is defined on the configuration xml. Now, regarding the configuration files outside the Ambari root… that’s completely possible, BUT be careful about your expectations. Remember that you are defining what the Python scripts do, so technically you can do anything. As long as you know where your files are, feel free to load them and use them inside the scripts. But since this is a distributed environment managed by Ambari, anything else you do manually (that is inside a python script) will obviously not get propagated to the nodes. So, suppose you load a file from /tmp/. The question is which node’s /tmp/. So in summary, you can do anything as long as you acknowledge what runs when and where. On the other side, if you stick to what Ambari gives you, then it will make sure configs get propagated and handles get called.

          One last note, your scripts and configuration files get installed on the Master and the Slaves when the Stack is installed. Afterwards, they also get installed if a new node is added. The ambari-agent is the one in charge of running your scripts and it is within the ambari-agent root that your files live. I think there’s a comment above where I also talk about this.

          • Ankit 2 years ago

            Thanks again..! In my case, I must expose as configurable, only the property-items specific to my application (and not the properties that are cluster specific). They are stored only in one single host and so it shouldn’t be an issue. Those property-items, however, are inside files like ‘abc.xml’ and ‘abc.properties’ . The XML items should be easy to list, as that’s the default implementation. However I am a little worried about how to make the items in abc.properties configurable via the UI. The format of these files are, as I mentioned, simple key-value pairs (like ‘key1=value1’). Please suggest some ideal way to achieve this if possible.

            Best Regards 🙂

          • Author
            Abdiel Aviles 2 years ago

            Here’s an un-elegant way to do it.

            On your-service-config-file.xml add a new variable that would host the contents of your file:


            <property>
            <name>your_properties_template</name>
            <value>line1=foo
            line2=bar
            line3=narf
            </value>
            <description>stuff</description>
            </property>

            Then inside your start handle you can dump the contents of this configuration property as follows:

            File("/path/to/your.properties",
            owner=params.your_user,
            content=InlineTemplate(params.your_properties_template)
            )

            Now, every time you edit your_properties_template on the Ambari dashboard, you’ll be effectively updating your.properties file.

            NOTE: An InlineTemplate is way more powerful that the example above. You don’t really need to use it as stated above but it’s good to have it in case you need to actually inject Ambari variables like host names or other configuration variables into your template.

  15. shashank rai 2 years ago

    Is there any way to store user entered password as a service configuration in encrypted/encoded form in ambari DB.?
    I have tried but it gets stored as a plain text.
    How Hive & oozie stored DB password in ambari DB?

    • Bijoyeta 2 years ago

      This configuration will help you:

      sample.password

      PASSWORD
      Sample password. Must be at least 6 characters

      • shashank rai 2 years ago

        I am using mentioned configuration only, but while entering password it shows in dotted form.But once ambari persist your service configuration in ambari DB it stored that password in a clear text.
        select * from serviceconfig;
        select * from serviceconfigmapping;
        select * from clusterconfig WHERE config_id= config_id_for_ur_service

    • Bijoyeta 2 years ago

      This configuration will help you: ”
      vnc.password

      PASSWORD
      VNC password. Must be at least 6 characters

      • Bijoyeta 2 years ago

        The XML configuration is not being loaded properly in the comments. Request the moderator to kindly delete my recent replies.

  16. shashank rai 2 years ago

    Ambari stores the service configuration in postgresql DB.
    My service has password property as a part of configuration file, i want it to be stored in enrypted/encoded form.Is there any way to do that?
    I have checked in Ambari DB but it stored in plain text for my service.
    How Hive & oozie stored their DB credential ?

  17. Bijoyeta 2 years ago

    Hi
    I have been through this article numerous times already. This is my seventh day and although I have been able to start and stop my custom application, I have still not been able to figure out a couple of stuff:
    1. How have you or the people above managed to ‘save’ your configuration files 🙁 I am wondering how to ‘re-configure my service files when starting the service’. What code do I write for that? After changing confs through the UI, the actual files inside ‘Configuration’ directory still have the old confs. I have the following code in my start(self, env) method:
    self.configure(env)
    and this in my configure(self, env) method:
    myconfigenv_tmpl = config['configurations']['myconfigenv']['content']
    File("/path/to/stack/Configuration/my-config-env.xml",
    content=InlineTemplate( myconfigenv_tmpl)
    )

    but none of them seem to be working. I know I am missing something here but there’s just not enough documentation to help. Please guide me
    2. What code would load external xml files located outside the config folder?
    The following code I have added in metainfo.xml:
    /path/to/my/ext/dir/configxxx
    And I have the file ‘config.xml’ in the directory /path/to/my/ext/dir , but none of the properties inside this file are displayed in the UI. I get a heading ‘Custom configxxx’ and below that an option to ‘Add Property’.
    Any help would be appreciated.
    Warm Regards
    Bijoyeta

    • shashank rai 2 years ago

      answer to your 1st question:-
      Whatever files exist under configuration is just a templates, the property values won’t be change in its file.
      ambari store property values for service configuration in ambari DB i.e postgresql by default.
      Follow the below command;-
      psql “user=ambari host=localhost password=bigdata”
      select * from serviceconfig; //In serviceconfig table you can see, your service name with serviconfig_id
      select * from serviceconfigmapping; // from here you’ll get config_id for your service
      select * from clusterconfig WHERE config_id=150 ; // this will show your service configuration.

  18. Gaurav 2 years ago

    Hi Abdiel,

    Thanks for this wonderful post. I am a newbie in Hadoop and have a requirement to install plain hadoop (not HDP) using Ambari. I think it will be possible by installing/adding various services viz. HDFS, Hive etc. using Ambari WebUI and following the steps as you mentioned above.

    Pardon my ignorance but I have the following basic questions:
    1. Please let me know if my understanding is correct?
    2. Also, I believe I would need to use the services code as mentioned in below github location to install them.

    https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services

    I believe these are the base codes for services and not the HDP customized ones. Please let me know if I am correct.

    Thanks
    Gaurav

    • Author
      Abdiel Aviles 2 years ago

      Just follow the Ambari Installation guide. Once you have Ambari installed, login and follow the wizard. After selecting your nodes Ambari will ask you which HDP version to install. HDP is Hortonworks’ Hadoop distribution. Once you have your HDP setup you will have your hadoop cluster setup. You don’t have to add any additional services if you don’t need to.

      • Gaurav 2 years ago

        Well the problem is, I don’t want to install HDP but plain vanilla Hadoop(It’s not that I dislike HDP but I want to get my hands on dirty on plain hadoop).
        Post Ambari installation, how should I move ahead to install plain hadoop.

        Any help would be greatly appreciated..

  19. Gaurav 2 years ago

    Well the problem is, I don’t want to install HDP but plain vanilla Hadoop(It’s not that I dislike HDP but I want to get my hands on dirty on plain hadoop).
    Post Ambari installation, how should I move ahead to install plain hadoop.

    Any help would be greatly appreciated..

  20. Gitan 1 year ago

    My service relies on zookeeper, so I would like Ambari to automatically generate the zk quorum rather than having to hardcode it in my configuration files. I see that hbase does something similar, but I cannot figure out exactly how it works. If I look in configuration/hbase-site.xml:

    hbase.zookeeper.quorum
    localhost
    Comma separated list of servers in the ZooKeeper Quorum.
    For example, “host1.mydomain.com,host2.mydomain.com,host3.mydomain.com”.
    By default this is set to localhost for local and pseudo-distributed modes
    of operation. For a fully-distributed setup, this should be set to a full
    list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
    this is the list of servers which we will start/stop ZooKeeper on.

    But then when I go to install Hbase, I see that it has somehow figured out the my zookeeper quorum is node1,node2, and node3. What would the best way to go about this be?

Leave a reply

Your email address will not be published. Required fields are marked *

*