Practical Data Science with Hadoop® and Spark: Designing and Building Effective Analytics at Scale

Practical Data Science with Hadoop® and Spark: Designing and Building Effective Analytics at Scale

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Book description

The Complete Guide to Data Science with Hadoop—For Technical Professionals, Businesspeople, and Students

Demand is soaring for professionals who can solve real data science problems with Hadoop and Spark. Practical Data Science with Hadoop® and Spark is your complete guide to doing just that. Drawing on immense experience with Hadoop and big data, three leading experts bring together everything you need: high-level concepts, deep-dive techniques, real-world use cases, practical applications, and hands-on tutorials.

The authors introduce the essentials of data science and the modern Hadoop ecosystem, explaining how Hadoop and Spark have evolved into an effective platform for solving data science problems at scale. In addition to comprehensive application coverage, the authors also provide useful guidance on the important steps of data ingestion, data munging, and visualization.

Once the groundwork is in place, the authors focus on specific applications, including machine learning, predictive modeling for sentiment analysis, clustering for document analysis, anomaly detection, and natural language processing (NLP).

This guide provides a strong technical foundation for those who want to do practical data science, and also presents business-driven guidance on how to apply Hadoop and Spark to optimize ROI of data science initiatives.

Show and hide more Table of contents Product information

Table of contents

  1. About This E-Book
  2. Title Page
  3. Copyright Page
  4. Contents
  5. Foreword
  6. Preface
    1. Focus of the Book
    2. Who Should Read This Book
    3. How to Use This Book
    4. Book Conventions
    5. Accompanying Code
    1. 1. Introduction to Data Science
      1. What Is Data Science?
      2. Example: Search Advertising
      3. A Bit of Data Science History
        1. Statistics and Machine Learning
        2. Innovation from Internet Giants
        3. Data Science in the Modern Enterprise
        1. The Data Engineer
        2. The Applied Scientist
        3. Transitioning to a Data Scientist Role
        4. Soft Skills of a Data Scientist
        1. Ask the Right Question
        2. Data Acquisition
        3. Data Cleaning: Taking Care of Data Quality
        4. Explore the Data and Design Model Features
        5. Building and Tuning the Model
        6. Deploy to Production
        1. Big Data—A Driver of Change
          1. Volume: More Data Is Now Available
          2. Variety: More Data Types
          3. Velocity: Fast Data Ingest
          1. Product Recommendation
          2. Customer Churn Analysis
          3. Customer Segmentation
          4. Sales Leads Prioritization
          5. Sentiment Analysis
          6. Fraud Detection
          7. Predictive Maintenance
          8. Market Basket Analysis
          9. Predictive Medical Diagnosis
          10. Predicting Patient Re-admission
          11. Detecting Anomalous Record Access
          12. Insurance Risk Analysis
          13. Predicting Oil and Gas Well Production Levels
          1. What Is Hadoop?
            1. Distributed File System
            2. Resource Manager and Scheduler
            3. Distributed Data Processing Frameworks
            1. Apache Sqoop
            2. Apache Flume
            3. Apache Hive
            4. Apache Pig
            5. Apache Spark
            6. R
            7. Python
            8. Java Machine Learning Packages
            1. Cost Effective Storage
            2. Schema on Read
            3. Unstructured and Semi-Structured Data
            4. Multi-Language Tooling
            5. Robust Scheduling and Resource Management
            6. Levels of Distributed Systems Abstractions
            7. Scalable Creation of Models
            8. Scalable Application of Models
            1. 4. Getting Data into Hadoop
              1. Hadoop as a Data Lake
              2. The Hadoop Distributed File System (HDFS)
              3. Direct File Transfer to Hadoop HDFS
              4. Importing Data from Files into Hive Tables
                1. Import CSV Files into Hive Tables
                1. Import CSV Files into HIVE Using Spark
                2. Import a JSON File into HIVE Using Spark
                1. Data Import and Export with Sqoop
                2. Apache Sqoop Version Changes
                3. Using Sqoop V2: A Basic Example
                1. Using Flume: A Web Log Example Overview
                1. Why Hadoop for Data Munging?
                2. Data Quality
                  1. What Is Data Quality?
                  2. Dealing with Data Quality Issues
                  3. Using Hadoop for Data Quality
                  1. Choosing the “Right” Features
                  2. Sampling: Choosing Instances
                  3. Generating Features
                  4. Text Features
                  5. Time-Series Features
                  6. Features from Complex Data Types
                  7. Feature Manipulation
                  8. Dimensionality Reduction
                  1. Why Visualize Data?
                    1. Motivating Example: Visualizing Network Throughput
                    2. Visualizing the Breakthrough That Never Happened
                    1. Comparison Charts
                    2. Composition Charts
                    3. Distribution Charts
                    4. Relationship Charts
                    1. R
                    2. Python: Matplotlib, Seaborn, and Others
                    3. SAS
                    4. Matlab
                    5. Julia
                    6. Other Visualization Tools
                    1. 7. Machine Learning with Hadoop
                      1. Overview of Machine Learning
                      2. Terminology
                      3. Task Types in Machine Learning
                      4. Big Data and Machine Learning
                      5. Tools for Machine Learning
                      6. The Future of Machine Learning and Artificial Intelligence
                      7. Summary
                      1. Overview of Predictive Modeling
                      2. Classification Versus Regression
                      3. Evaluating Predictive Models
                        1. Evaluating Classifiers
                        2. Evaluating Regression Models
                        3. Cross Validation
                        1. Model Training
                        2. Batch Prediction
                        3. Real-Time Prediction
                        1. Tweets Dataset
                        2. Data Preparation
                        3. Feature Generation
                        4. Building a Classifier
                        1. Overview of Clustering
                        2. Uses of Clustering
                        3. Designing a Similarity Measure
                          1. Distance Functions
                          2. Similarity Functions
                          1. k-means Clustering
                          2. Latent Dirichlet Allocation
                          1. Data Ingestion
                          2. Feature Generation
                          3. Running Latent Dirichlet Allocation
                          1. Overview
                          2. Uses of Anomaly Detection
                          3. Types of Anomalies in Data
                          4. Approaches to Anomaly Detection
                            1. Rules-based Methods
                            2. Supervised Learning Methods
                            3. Unsupervised Learning Methods
                            4. Semi-Supervised Learning Methods
                            1. Data Ingestion
                            2. Building a Classifier
                            3. Evaluating Performance
                            1. Natural Language Processing
                              1. Historical Approaches
                              2. NLP Use Cases
                              3. Text Segmentation
                              4. Part-of-Speech Tagging
                              5. Named Entity Recognition
                              6. Sentiment Analysis
                              7. Topic Modeling
                              1. Small-Model NLP
                              2. Big-Model NLP
                              1. Bag-of-Words
                              2. Word2vec
                              1. Stanford CoreNLP
                              2. Using Spark for Sentiment Analysis
                              1. Automated Data Discovery
                              2. Deep Learning
                              3. Summary
                              1. Quick Command Dereference
                                1. General User HDFS Commands
                                2. List Files in HDFS
                                3. Make a Directory in HDFS
                                4. Copy Files to HDFS
                                5. Copy Files from HDFS
                                6. Copy Files within HDFS
                                7. Delete a File within HDFS
                                8. Delete a Directory in HDFS
                                9. Get an HDFS Status Report (Administrators)
                                10. Perform an FSCK on HDFS (Administrators)
                                1. General Hadoop/Spark Information
                                2. Hadoop/Spark Installation Recipes
                                3. HDFS
                                4. MapReduce
                                5. Spark
                                6. Essential Tools
                                7. Machine Learning
                                Show and hide more

                                Product information

                                • Title: Practical Data Science with Hadoop® and Spark: Designing and Building Effective Analytics at Scale
                                • Author(s): Ofer Mendelevitch, Casey Stella, Douglas Eadline
                                • Release date: December 2016
                                • Publisher(s): Addison-Wesley Professional
                                • ISBN: 9780134029733

                                You might also like

                                Check it out now on O’Reilly

                                Dive in for free with a 10-day trial of the O’Reilly learning platform—then explore all the other resources our members count on to build skills and solve problems every day.