Saturday, 1 June 2013

Why shared mutable state is the root of all evil.

Introduction


A while back I went to a talk by Martin Odersky introducing Scala. Martin is the lead designer of Scala and a very smart guy. You can have a look at the slides from the talk here.

He described how the effect of Moore's law (that the number of transistors on a reasonable priced CPU doubles roughly every 2 years) doesn't lead to higher clock speeds anymore since the early 2000s. The number of transistors is still doubling but the extra transistors are used to include additional cores per CPU and to increase L2 and L3 cache sizes. Before the 2000s programmers got a free lunch meaning that the performance of their programs automatically improved with each new processor generation. This free lunch is now over and the performance of a single threaded program hasn't improved much over the last 10 years. To be able to create programs with smaller latency or higher throughput parallel programming mechanism are needed. 

What I found really profound was this pseudo equation which Martin introduced:

       non-determinism = parallel processing + mutable state

This equation basically means that both parallel processing and mutable state combined result in non-deterministic program behaviour. If you just do parallel processing and have only immutable state everything is fine and it is easy to reason about programs. On the other hand if you want to do parallel processing with mutable data you need to synchronize the access to the mutable variables which essentially renders these sections of the program single threaded. This  is not really new but I haven't seen this concepts expressed so elegantly. A non-deterministic program is broken.

Why are parallel programs without proper synchronization broken?

There are a few different things which can go wrong when no proper synchronization policy is enforced: Race conditions, visibility issues and out-of-order processing.

Race condition

A race condition means that the outcome of an algorithm is depend on its timing and lack of influences from other threads. Lets have a lock at the check-then-act race condition. In Java it would look something like this:

public class BrokenMap<E, V> extends HashMap<E ,V> {
    public T putIfAbsent(final E key, final T value) {
        if (!this.containsKey(key)) {
            return this.put(key, value);
        } else {
            return this.get(key);
        }
    )
}

In the putIfAbsent method we first check if the element is part of the map already. If this is not the case we add the element. Otherwise we just return the existing element. This code works perfectly fine in a single threaded environment. In a multithreaded environment however a second thread could have jumped in between the two calls to containsKey and put and put its value their first. Once thread one is continuing execution again it would override the already existing value of thread two in the map.

For the method putIfAbsent to be thread-safe it needs to guaranteed that the two calls to containsKey and put are always run uninterrupted. They need to be atomic. The easiest way to ensure this to put both calls into a synchronized block which essentially renders the program single threaded in this section.

Another form of race condition is read-modify-write. In Java it would look something like this:


public class BrokenCounter {
    private long count = 0;
    public long increment() {
        return ++count;
    }
}

The class BrokenCounter has a method increment. This class is not thread-safe because the call to increment the counter ++count is not just one operation. Under the hood it is actually 3 operations.
  1. Read the old value of count.
  2. Increment the value of count.
  3. Store the new value of count.
If a second thread interrupts the first thread somewhere between step 1 and 3 it would read the same value as thread one, increment it by one and store it back into the count variable. Once the first thread resumes it would also increment the old value and store it back into the count variable. The result of this would be a counter which has been only incremented by one even though it was called twice.

For the method increment to be thread-safe it needs to guarantee that calls to ++count will not be interrupted. The easiest way to achieve this is by putting the call into a synchronized block which essentially renders this section of the program single threaded.


Visibility

Visibility means essentially that an action in thread one is visible to a second thread. In a single threaded program when you write a new value to a variable it is guaranteed that all future reads to this variable will return the new value. In a multithreaded program such a guarantee doesn't exist. Lets have a look at the example program BrokenServer:

public class BrokenServer {

    private boolean stopped=false;
    
    public void run() {
        while(!stopped) {
            /* do something useful */
        }
    }
 
    public void cancel() {
        this.stopped = true;
    }
}


In this program a method run keeps processing stuff until it get interrupted by another thread calling the cancel method. It is a bid odd to imagine but the thread in the run method may actually never see the updated value of stopped. This is because the variable stopped may be cached in a local variable in the JVM or the value is resolved from the L1/L2 cache of the core rather than from main memory. To fix this we need to tell the JVM to include a memory barrier after writing the stopped variable and another one before reading the stopped variable. The memory barrier guarantees that all writes to the variable happen before all successive reads from the same variable (like you would expect in a single threaded program).

The easiest way to achieve this is to use a synchronized block around reading and writing the variable stopped which once again renders this part of the program single threaded.


Ordering


If you made it this far you must be truly interested in Java Concurrency. Lets have a look at one more case which is even more weird than the visibility issue above. The JVM and modern CPUs are allowed to execute code out of order if that helps with speeding up the code or improve memory access pattern.

Lets have a look at another piece of code:

public class BrokenOrder {
    int a = 0;
    int b = 0;
 
    public void assignment() {
        a = 1;
        b = 2;
    }
 
    public int getSum() {
        return a+b;
    } 
}
It is actually surprisingly difficult to reason about this very simple program. What would be the result of getSum if called in a multithreaded environment?

  • Returns 0 if getSum runs before assignment
  • Returns 3 if getSum runs after assignment
  • Returns 1 if getSum runs after a=2 assignment but before b=3 assignment
  • Returns 2 if getSum runs after b=2 assignment but before a=2 assignment

This last case can happen if the JVM / CPU decides to process the assignment method out of order. To prohibit this kind of non-deterministic behavior once again we need to use a memory barrier to prevent out-of-order processing. The easiest way to do this in Java is to use a synchronized block which renders the program single threaded in this section.

Conclusion


Shared mutable state and parallel processing doesn't go together at all. If you do not use proper synchronization you will create extremely difficult to find bugs and your programs are basically broken even if they appear to work just fine in most cases.

1,218 comments:

  1. Very well explained. I could able to understand why people again started moving to Functional Programming.

    ReplyDelete
    Replies
    1. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us

      You will get an introduction to the Python programming language and understand the importance of it. How to download and work with Python along with all the basics of Anaconda will be taught. You will also get a clear idea of downloading the various Python libraries and how to use them.
      Topics
      About [url=https://www.excelr.com/data-science-certification-course-training-in-singapore]Excelr Solutions[/url] and Innodatatics
      Introduction to Python
      Installation of Anaconda Python
      Difference between Python2 and Python3
      Python Environment
      Operators
      Identifiers
      Exception Handling (Error Handling)

      Delete
    2. This comment has been removed by the author.

      Delete
  2. Nice explanation, but your last example is confusing because there is no b=3 or a=2 assignments.

    ---
    Returns 1 if getSum runs after a=2 assignment but before b=3 assignment
    Returns 2 if getSum runs after b=2 assignment but before a=2 assignment

    ReplyDelete
  3. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training India . Nowadays Java has tons of job opportunities on various vertical industry.

    ReplyDelete
  4. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
    Best Devops Training in pune
    Data science training in Bangalore

    ReplyDelete
  5. Read all the information that i've given in above article. It'll give u the whole idea about it.
    Best Devops Training in pune
    Data science training in Bangalore

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Really you have done great job,There are may person searching about that now they will find enough resources by your post
    Python Online training
    python Training in Chennai
    Python training in Bangalore

    ReplyDelete
  8. I appreciate your efforts because it conveys the message of what you are trying to say. It's a great skill to make even the person who doesn't know about the subject could able to understand the subject . Your blogs are understandable and also elaborately described. I hope to read more and more interesting articles from your blog. All the best.
    Selenium training in Chennai
    Selenium training in Bangalore
    Selenium training in Pune
    Selenium Online training

    ReplyDelete
  9. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

    DevOps is currently a popular model currently organizations all over the world moving towards to it. Your post gave a clear idea about knowing the DevOps model and its importance.

    Good to learn about DevOps at this time.


    devops training in chennai | devops training in chennai with placement | devops training in chennai omr | devops training in velachery | devops training in chennai tambaram | devops institutes in chennai | devops certification in chennai | trending technologies list 2018

    ReplyDelete
  10. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
    Thanks & Regards,
    VRIT Professionals,
    No.1 Leading Web Designing Training Institute In Chennai.

    And also those who are looking for
    Web Designing Training Institute in Chennai
    SEO Training Institute in Chennai
    Photoshop Training Institute in Chennai
    PHP & Mysql Training Institute in Chennai
    Android Training Institute in Chennai

    ReplyDelete
  11. Really awesome blog. Your blog is really useful for me
    Regards,
    best selenium training institute in chennai | selenium course in chennai

    ReplyDelete
  12. Nice post!Everything about the future(học toán cho trẻ mẫu giáo) is uncertain, but one thing is certain: God has set tomorrow for all of us(toán mẫu giáo 5 tuổi). We must now trust him and in this regard, you must be(cách dạy bé học số) very patient.

    ReplyDelete
  13. I Regreat For Sharing The information The InFormation shared Is Very Valuable Please Keep Updating Us Time Just Went On Reading The Article Python Online Training AWS Online Training Hadoop Online Training Data Science Online Training

    ReplyDelete
  14. Great!it is really nice blog information.after a long time i have grow through such kind of ideas.Thanks for share your thoughts with us.
    Selenium Training in Chennai | SeleniumTraining Institute in Chennai

    ReplyDelete
  15. Time is free but it's priceless(khóa học toán tư duy) . You cannot own it, but you can use it(cách dạy bé học số) . You can use it, but you can't keep it(toán tư duy logic là gì). Once you lose it, you will not be able to get it back.

    ReplyDelete
  16. Thanks for sharing valuable information. Your blogs were helpful to Azure learners. I request to update the blog through step-by-step. Also, find the Azure news at

    Data Science Course

    ReplyDelete
  17. Indeed, the goal of Spring Boot is not to provide new solutions for the many problem domains already solved.

    spring boot tutorial

    ReplyDelete
  18. Thanks for sharing valuable article having good information and also gain worthful knowledge.

    Oracle Integration cloud service online training

    ReplyDelete
  19. Thank you for sharing your awesome and valuable article this is the best blog for the students they can also learn.

    Workday HCM Online Training

    ReplyDelete
  20. Thanks for sharing such a good article having valuable information.best to learn Big Data and Hadoop Training course.

    Big Data and Hadoop Training In Hyderabad

    ReplyDelete
  21. Its very informative blog and useful article thank you for sharing with us , keep posting learn
    Data Science online Course


    dot NET Course

    ReplyDelete
  22. It is a great post. Keep sharing such kind of useful information.

    Education
    Technology

    ReplyDelete
  23. thanks for sharing such a nice info.I hope you will share more information like this. please keep on sharing!
    AWS Online Training
    AWS Training in Hyderabad
    Amazon Web Services Online Training

    ReplyDelete
  24. And indeed, I’m just always astounded concerning the remarkable things served by you. Some four facts on this page are undeniably the most effective I’ve had.
    fire and safety course in chennai
    safety course in chennai

    ReplyDelete
  25. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  26. An amazing web journal I visit this blog, it's unbelievably wonderful. Oddly, in this blog's content made without a doubt and reasonable. The substance of data is informative.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  27. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  28. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

    Android Interview Questions and Answers


    Angular JS Interview Questions and Answers

    ReplyDelete
  29. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  30. Thanks For Sharing The Information The Information Shared Is Very Valuable Please Keep Updating Us Time Just Went On Reading The article Python Online Course Hadoop Online Course Aws Online Course Data Science Online Course

    ReplyDelete
  31. Thanks for providing a useful article containing valuable information. start learning the best online software courses.

    Workday Online Training

    ReplyDelete
  32. Appericated the efforts you put in the content of Data Science .The Content provided by you for Data Science is up to date and its explained in very detailed for Data Science like even beginers can able to catch.Requesting you to please keep updating the content on regular basis so the peoples who follwing this content for Data Science can easily gets the updated data.
    Thanks and regards,
    Data Science training in Chennai
    Data Science course in chennai with placement
    Data Science certification in chennai
    Data Science course in Omr

    ReplyDelete
  33. Such a useful article. The efforts you had made for writing this awesome article are great.

    ExcelR Data Science Course Bangalore

    ReplyDelete
  34. Such a wonderful blog on Machine learning . Your blog have almost full information about Machine learning .Your content covered full topics of Machine learning that it cover from basic to higher level content of Machine learning . Requesting you to please keep updating the data about Machine learning in upcoming time if there is some addition.
    Thanks and Regards,
    Machine learning tuition in chennai
    Machine learning workshops in chennai
    Machine learning training with certification in chennai

    ReplyDelete
  35. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.
    date analytics certification training courses

    ReplyDelete
  36. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete

  37. Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time.

    Big Data Course

    ReplyDelete
  38. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!data science course in dubai

    ReplyDelete
  39. Thanks for the informative and helpful post, obviously in your blog everything is good..
    Data Science Course in Pune

    ReplyDelete
  40. I feel strongly about this and so really like getting to know more on this kind of field.
    Do you mind updating your blog post with additional insight?
    It should be really useful for all of us.
    Digital marketing service in sehore
    website designer in sehore

    ReplyDelete
  41. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  42. Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.data science course in dubai

    ReplyDelete
  43. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  44. Really I Appreciate The Effort You Made To Share The Knowledge. This Is Really A Great Stuff For Sharing. Keep It Up . Thanks ForQuality posts is the crucial to invite the visitors to visit the web page, that's what this web page is providing. data science course in singapore

    ReplyDelete
  45. Great blog created by you. I read your blog, its best and useful information.
    Digital Marketing Training
    ServiceNow Online Training
    EDI Online Training

    ReplyDelete
  46. Website Planning Best institute for digital marketing course in delhi. Initialisation of Digital Marketing.. Website Creation. Content Writing. Search Engine Optimization. Local Seo. Google Webmaster. Bing Webmaster.
    Digital Marketing training in Laxmi Nagar

    ReplyDelete
  47. Just now I read your blog, it is very helpful nd looking very nice and useful information.
    Digital Marketing Online Training
    Servicenow Online Training
    EDI Online Training

    ReplyDelete
  48. Thanks for this post, I really appriciate. I have read posts,
    all are in working condition. and I really like your writing style.
    autocad in bhopal
    3ds max classes in bhopal
    CPCT Coaching in Bhopal
    java coaching in bhopal
    Autocad classes in bhopal
    Catia coaching in bhopal

    ReplyDelete
  49. UV Gullas College of MedicineUV Gullas College of Medicine- Do you Want to do MBBS in Philippines? Then make your decision with us.! Here no need any entrance examination.

    ReplyDelete
  50. On the off chance that you are searching with the expectation of complimentary logos for your business, there are a few ways to deal with getting one for yourself. Actually, a standout amongst the best choices is to get it through 'across the board' kind of bundles by expert logo configuration firms, where you buy something at a focused cost and get your logo intended for nothing. logo design service

    ReplyDelete
  51. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  52. One of the world's premier academic and research institutions, the UV Gullas College of Medicine has driven new ways of thinking since our 1919 founding. We offer students high quality teaching and research in a safe and friendly setting for their studies, the perfect place to learn and grow.

    ReplyDelete
  53. HVO Vietnam Education Services Joint Stock Company was established in 2017 with the desire to bring a learning environment to maximize the creativity and logic of children through the subjects of the program. Home thinking math (Toán tư duy tại nhà ), Army summer camp (trại hè quân đội), Homeschool

    ReplyDelete
  54. MBBS in AbroadGet MBBS in Abroad for low fees @ UV Gullas College of Medicine in Philippines. We are one of the best MBBS Universities in abroad.

    ReplyDelete
  55. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
    www.technewworld.in
    How to Start A blog 2019
    Eid AL ADHA

    ReplyDelete
  56. Thank you for excellent article.You made an article that is interesting.
    Tavera car for rent in chennai|Indica car for rent in chennai|innova car for rent in chennai|mini bus for rent in chennai|tempo traveller for rent in chennai
    Keep on the good work and write more article like this...

    Great work !!!!Congratulations for this blog


    ReplyDelete
  57. vlsa global services providing best ece project in chennai.VLSA Global Services is the best
    ece projects in chennai, VLSA Global Services offers ece projects in Chennai and IEEE 2013 Final Year projects for Engineering students in JAVA, Dot Net, Android, Oracle, matlab, embedded system, python and PHP technologies

    ReplyDelete
  58. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us
    You will get an introduction to the Python programming language and understand the importance of it. How to download and work with Python along with all the basics of Anaconda will be taught. You will also get a clear idea of downloading the various Python libraries and how to use them.
    Topics
    About ExcelR Solutions and Innodatatics
    Do's and Don’ts as a participant
    Introduction to Python
    Installation of Anaconda Python
    Difference between Python2 and Python3
    Python Environment
    Operators
    Identifiers
    Exception Handling (Error Handling)
    Excelr Solutions

    ReplyDelete
  59. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us
    You will get an introduction to the Python programming language and understand the importance of it. How to download and work with Python along with all the basics of Anaconda will be taught. You will also get a clear idea of downloading the various Python libraries and how to use them.
    Topics
    About ExcelR Solutions and Innodatatics
    Do's and Don’ts as a participant
    Introduction to Python
    Installation of Anaconda Python
    Difference between Python2 and Python3
    Python Environment
    Operators
    Identifiers
    Exception Handling (Error Handling)
    Excelr Solutions

    ReplyDelete
  60. There is an ever-growing large community of Indian students who study at our campus to realize their cherished dreams of becoming a doctor. Lyceum Northwestern University is committed to making our students become compassionate, knowledgeable, highly-competent physicians by providing them with early hands-on clinical training and opportunities to hone their professional skills.

    ReplyDelete
  61. Thanks for sharing such a wonderful blog on Python .This blog contains so much data about Python ,like if anyone who is searching for the Python data will easily grab the knowledge of Python from this.Requested you to please keep sharing these type of useful content so that other can get benefit from your shared content.
    Thanks and Regards,
    Top Institutes for Python in Chennai.
    Best Python institute in Chennai .
    Python course in chennai .

    ReplyDelete
  62. Just seen your Article, it amazed me and surpised me with god thoughts that eveyone will benefit from it. It is really a very informative post for all those budding entreprenuers planning to take advantage of post for business expansions. You always share such a wonderful articlewhich helps us to gain knowledge .Thanks for sharing such a wonderful article, It will be deinitely helpful and fruitful article.
    Thanks
    DedicatedHosting4u.com

    ReplyDelete
  63. Such a great and informative article.
    You just made my day thanks for sharing this article.

    data science course singapore is the best data science course

    ReplyDelete
  64. I like it very much. it's very good and relevant information. good content, and unique design.Great article thanks for sharing us. ASP.NET is a web application structure created and advertised by Microsoft to enable software engineers to fabricate dynamic sites.
    anyone want to learn advance develops training
    visit: Asp.net Training in Chennai

    ReplyDelete
  65. Excellent Blog. I really want to admire the quality of this post. I like the way of your presentation of ideas, views and valuable content. No doubt you are doing great work. I’ll be waiting for your next post. Thanks .Keep it up! Kindly visit us @Luxury Boxes
    Premium Packaging
    Luxury Candles Box
    Earphone Packaging Box
    Wireless Headphone Box
    Innovative Packaging Boxes
    Wedding gift box
    Leather Bag Packaging Box
    Cosmetics Packaging Box
    Luxury Chocolate Boxes

    ReplyDelete
  66. One of the world's premier academic and research institutions, the UV gullas medical college has driven new ways of thinking since our 1919 founding. We offer students high quality teaching and research in a safe and friendly setting for their studies, the perfect place to learn and grow.

    ReplyDelete
  67. This is a nice Site to watch out for and we provided information on
    vidmate make sure you can check it out and keep on visiting our Site.

    ReplyDelete
  68. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    pmp certification course

    ReplyDelete


  69. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing....

    software Providers
    erp software
    crm software
    best software company in chennai
    software company in india
    Top software company in chennai

    ReplyDelete
  70. This is really a big and great source of information. We can all contribute and benefit from reading as well as gaining knowledge from this content. Just amazing
    experience. Thanks for sharing such nice information.
    Event Management in Pondicherry | Wedding Decorators in Trichy | Wedding Photographers in Trichy | Wedding Planner in Pondicherry | Wedding Decorators in Pondicherry | Candid Photography Pondicherry | Wedding Photographers in Pondicherry

    ReplyDelete
  71. Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows.

    ReplyDelete
  72. This comment has been removed by the author.

    ReplyDelete
  73. Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows.

    ReplyDelete
  74. Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows.

    ReplyDelete
  75. Thank you for your post. This is superb information. It is amazing and great to visit your site.

    data science certification course training

    ReplyDelete
  76. Дээд чанар бол зүгээр л( đá ruby thiên nhiên ) санаатай биш юм. Энэ нь өндөр( đá ruby nam phi ) түвшний төвлөрөл, тусгай хүчин( Đá Sapphire ) чармайлт, ухаалаг ( đá sapphire hợp mệnh gì )чиг баримжаа, чадварлаг туршлага, ( đá ruby đỏ )саад тотгорыг даван туулах( bán đá sapphire thô ) боломжийг хардаг.

    ReplyDelete
  77. This comment has been removed by the author.

    ReplyDelete
  78. Thanks for Sharing this Great information worth reading this article. Leverage SharePoint Features from veelead solutions

    ReplyDelete
  79. I am really enjoying reading your well-written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
    PHP Training in Coimbatore
    PHP Course in Coimbatore
    Selenium Training in Coimbatore
    seo course in coimbatore
    Android Training in Coimbatore
    Embedded course in Coimbatore
    German coaching classes in Coimbatore
    Java Training in Bangalore

    ReplyDelete
  80. Thanks for sharing such a wonderful blog on Python .This blog contains so much data about Python ,like if anyone who is searching for the Python data will easily grab the knowledge of Python from this.Requested you to please keep sharing these type of useful content so that other can get benefit from your shared content.
    Thanks and Regards,
    Top Institutes for Python in Chennai.
    Best Python institute in Chennai .
    Python course in chennai .

    ReplyDelete
  81. This comment has been removed by the author.

    ReplyDelete
  82. Download latest audio and video file fromvidmate

    ReplyDelete
  83. Download latest audio and video file fromvidmate

    ReplyDelete
  84. Thanks a lot for writting such a great article. It's really has lots of insights and valueable informtion.
    If you wish to get connected with AI world, we hope the below information will be helpful to you.
    Python Training Institute in Pune
    Python Interview Questions And Answers For Freshers
    Data -Science
    ML(Machine Learning) related more information then meet on EmergenTeck Training Institute .
    Machine Learning Interview Questions And Answers for Freshers
    Thank you.!

    ReplyDelete
  85. Download latest audio and video file fromvidmate

    ReplyDelete
  86. I was scrolling the internet like every day, there I found this article which is related to my interest. The way you covered the knowledge about the subject and the top builders in bhopal was worth to read, it undoubtedly cleared my vision and thoughts towards B 3 bhk flat in ayodhy bypass road . Your writing skills and the way you portrayed the examples are very impressive. The knowledge about 2 bhk flat in ayodhya bypaas road is well covered. Thank you for putting this highly informative article on the internet which is clearing the vision about top builders in Bhopal and who are making an impact in the real estate sector by building such amazing townships.

    ReplyDelete
  87. An overwhelming web journal I visit this blog, it's unfathomably amazing. Unusually, in this present blog's substance made inspiration driving truth and reasonable. The substance of data is enlightening
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  88. I love to read your article. Your writingstyle is too good
    ExcelR is a global leader delivering a wide gamut of management and technical training over 40 countries. We are a trusted training delivery partner of 350+ corporate clients and universities across the globe with 28,000+ professionals trained across various courses. With over 20 Franchise partners all over the world, ExcelR helps individuals and organisations by providing courses based on practical knowledge and theoretical concepts.

    Excelr Solutions

    ReplyDelete
  89. I love your article so much. Good job
    ExcelR is a global leader delivering a wide gamut of management and technical training over 40 countries. We are a trusted training delivery partner of 350+ corporate clients and universities across the globe with 28,000+ professionals trained across various courses. With over 20 Franchise partners all over the world, ExcelR helps individuals and organisations by providing courses based on practical knowledge and theoretical concepts.

    Excelr Solutions

    ReplyDelete
  90. amazing blog post. thanks for information.
    Home Salon

    ReplyDelete
  91. This blog contain alot of informative information.Has a good writing shkills too. Thank you for the information.
    python training in bangalore

    ReplyDelete
  92. Thanks for sharing this useful information.
    python training

    ReplyDelete
  93. Great article. Couldn’t be write much better!
    https://www.bisptrainings.com/Courses/Microsoft-Power-BI

    ReplyDelete
  94. I love your article so much. Good job
    Participants who complete the assignments and projects will get the eligibility to take the online exam. Thorough preparation is required by the participants to crack the exam. ExcelR's faculty will do the necessary handholding. Mock papers and practice tests will be provided to the eligible participants which help them to successfully clear the examination.

    Excelr Solutions

    ReplyDelete
  95. I love your article so much. Good job
    Participants who complete the assignments and projects will get the eligibility to take the online exam. Thorough preparation is required by the participants to crack the exam. ExcelR's faculty will do the necessary handholding. Mock papers and practice tests will be provided to the eligible participants which help them to successfully clear the examination.

    Excelr Solutions

    ReplyDelete
  96. Nice Presentation and its hopefull words..
    if you want a cheap web hosting in web
    cheap web hosting company chennai

    ReplyDelete
  97. thanks for ur valuable information,keep going touch with us

    Scaffolding dealers in chennai

    ReplyDelete
  98. An overwhelming web journal I visit this blog, it's unfathomably amazing. Unusually, in this present blog's substance made inspiration driving truth and reasonable. The substance of data is enlightening
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  99. If you are looking for Best Gyms in Ghaziabad then click on the given link.

    ReplyDelete
  100. Thanks for sharing such a wonderful blog on Mean Stack .This blog contains so much data about Mean Stack ,like if anyone who is searching for the Mean Stack data,They will easily grab the knowledge of from this.Requested you to please keep sharing these type of useful content so that other can get benefit from your shared content.
    Thanks and Regards,
    Mean Stack training in Chennai
    Best mean stack training in Chennai
    Top Mean stack raining in Chennai
    Course fees for Mean stack in Chennai
    Mean stack training fees in Velachery, Chennai

    ReplyDelete
  101. Thanks for sharing useful information.. we have learned so much information from your blog..... keep sharing
    Oracle Fusion HCM Online Training

    ReplyDelete
  102. While composing the last paper of your last B.Com test, a plenty of considerations experience your psyche. A great deal of these contemplations incorporate plans about resting for a whole week or celebrating for a whole week, contingent upon your individual inclinations. Be that as it may, trust us, none of that is really going to happen in light of the fact that when you complete your tests your folks, relatives, neighbours, and even your Facebook companions will begin getting some information about your feasible arrangements. What's more, don't mistake them for your gathering or dozing plans since they are alluding to your vocation Career after B com plans. In the present focused world, you are offered with many profession improving courses. On the off chance that you are not happy with the profession or course you decided for yourself at that point there are some present moment yet high worth – low speculation courses accessible in the market.

    ReplyDelete