How to Import CSV File Data into MongoDB Using Node.js and Express

Setup Folder

Open your command prompt and create a new folder using the following command:

mkdir csv

After creating the folder, change into it by running:

cd csv

Setup Node In Folder

Initialize the Node.js environment in your project:

npm init -y

This generates a package.json file containing the metadata for your application.

Install Packages

Install the required packages for handling CSV parsing, routing, template rendering, and file uploads:

npm install csvtojson express ejs mongoose multer body-parser

Add New Folders

Create the following folders in your project root directory:

  • models
  • public
  • views

Models Folder (csv.js)

Add a new file inside the models folder named csv.js to define your MongoDB collection schema:


var mongoose = require('mongoose');

var csvSchema = new mongoose.Schema({
    FirstName:{
        type: String
    },
    LastName:{
        type: String
    },
    SSN:{
        type: String
    },
    Test1:{
        type: Number
    },
    Test2:{
        type: Number
    },
    Test3:{
        type: Number
    },
    Test4:{
        type: Number
    },
    Final:{
        type: Number
    },
    Grade:{
        type: String
    }
});

module.exports = mongoose.model('studentsrecords', csvSchema);

Views Folder (demo.ejs)

Add a new file inside the views folder named demo.ejs to handle the file upload form and render imported student records:


<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
    <link rel="stylesheet" href="/css/bootstrap.min.css">
</head>
<body>
        <nav class="navbar navbar-expand-lg navbar-dark bg-primary fixed-top">
                <a class="navbar-brand" href="#">CsvToMongo</a>
              </nav>
    <div class="container">
        <div class=" nav justify-content-center" style="margin-top:100px;">
         <div class="card border-warning mb-3 " style="max-width: 20rem;">
                <div class="card-header"><h5>Upload csv file</h5></div>
                <div class="card-body">
                        <form action="/" method="post" enctype="multipart/form-data">
                            <input type="file" name="csv"><br><br>
                         <div class="text-center"><button type="submit" class="btn btn-lg btn-primary">submit</button></div>   
                        </form>
                </div>
         </div>
    </div><br>
    <%if(data){%>
    <div>
        <table class="table table-hover table-responsive table-striped  nav justify-content-center" style="width: auto"> 
                <thead>
                    <tr class="bg-primary">
                        <th>S.no</th>
                        <th style="padding-right: 1em">LastName</th>
                        <th style="padding-right: 1em">FirstName</th>
                        <th style="padding-right:2em;padding-left:2em;">SSN</th>
                        <th>Test1</th>
                        <th>Test2</th>
                        <th>Test3</th>
                        <th>Test4</th>
                        <th>Final</th>
                        <th>Grade</th>
                    </tr>
                </thead>
                <tbody style="overflow-x: scroll; height:350px;" class="table-bordered">
                    <%for(var i=0;i< data.length;i++){%>
                    <tr class="text-center">
                                  <td ><%= i+1%></td>
                                  <td style="padding-right: 1em"><%= data[i].LastName%></td>
                                  <td style="padding-left: 1em;"><%= data[i].FirstName%></td>
                                  <td style="padding-right:1em;padding-left:1em;"><%= data[i].SSN%></td>
                                  <td style="padding-left: 1em"><%= data[i].Test1%></td>
                                  <td style="padding-left: 1em"><%= data[i].Test2%></td>
                                  <td style="padding-left: 1em"><%= data[i].Test3%></td>
                                  <td style="padding-left: 1.2em"><%= data[i].Test4%></td>
                                  <td style="padding-left: 1.2em"><%= data[i].Final%></td>
                                  <td style="padding-left: 1.2em"><%= data[i].Grade%></td>
                    </tr>
                    <%}%>
                </tbody>
        </table>
    </div>
    <%}%>
<br>
</body>
</html>

Starting Point (app.js)

Create an app.js file in your root folder as the main entry point:


var express     = require('express');
var mongoose    = require('mongoose');
var multer      = require('multer');
var path        = require('path');
var csvModel    = require('./models/csv');
var csv         = require('csvtojson');
var bodyParser  = require('body-parser');

var storage = multer.diskStorage({
    destination:(req,file,cb)=>{
        cb(null,'./public/uploads');
    },
    filename:(req,file,cb)=>{
        cb(null,file.originalname);
    }
});

var uploads = multer({storage:storage});

//connect to db
mongoose.connect('mongodb://localhost:27017/csvdemos',{useNewUrlParser:true})
.then(()=>console.log('connected to db'))
.catch((err)=>console.log(err))

//init app
var app = express();

//set the template engine
app.set('view engine','ejs');

//fetch data from the request
app.use(bodyParser.urlencoded({extended:false}));

//static folder
app.use(express.static(path.resolve(__dirname,'public')));

//default pageload
app.get('/',(req,res)=>{
    csvModel.find((err,data)=>{
         if(err){
             console.log(err);
         }else{
              if(data!=''){
                  res.render('demo',{data:data});
              }else{
                  res.render('demo',{data:''});
              }
         }
    });
});

var temp ;

app.post('/',uploads.single('csv'),(req,res)=>{
//convert csvfile to jsonArray   
csv()
.fromFile(req.file.path)
.then((jsonObj)=>{
    console.log(jsonObj);
    for(var x=0;x<jsonObj.length;x++){
         temp = parseFloat(jsonObj[x].Test1)
         jsonObj[x].Test1 = temp;
         temp = parseFloat(jsonObj[x].Test2)
         jsonObj[x].Test2 = temp;
         temp = parseFloat(jsonObj[x].Test3)
         jsonObj[x].Test3 = temp;
         temp = parseFloat(jsonObj[x].Test4)
         jsonObj[x].Test4 = temp;
         temp = parseFloat(jsonObj[x].Final)
         jsonObj[x].Final = temp;
     }
     csvModel.insertMany(jsonObj,(err,data)=>{
            if(err){
                console.log(err);
            }else{
                res.redirect('/');
            }
     });
   });
});

//assign port
var port = process.env.PORT || 3000;
app.listen(port,()=>console.log('server run at port '+port));

Open your package.json and under "scripts" add "start": "node app.js".

Resources

Download Code From here

Watch Video Tutorial

Comments

Post a Comment

Popular posts from this blog

How to Setup a Flask Project in Python: Step-by-Step Guide

Image
Prerequisite Python What Is Flask ? Flask is a lightweight Web framework written in Python. It is developed by  Armin Ronacher. Flask is based on the Werkzeug WSGI toolkit and Jinja2 template engine Setup Development Environment First Install Virtualenv In Python virtualenv  is a virtual python environment builder.virtualenv    is used to manage python packages for different projects. Using virtualenv allows you to avoid installing python packages globally which could break system tools or other projects. Install the virtualenv using the following command.This command needs administrator privileges # pip install virtualenv After installing the virtualenv . Make Folder create new folder using the following command  # mkdir flaskdemo change to the current created directory/folder # cd flaskdemo Create a Virtual Environment for Flask In Folder Now we will create a virtualenv .  The commands below will c...
Read more

Hashing Password With Bcrypt In Node

Image
Image by Jae Rue from Pixabay Introduction In this article we will be using the bcryptjs javascript library for hashing and compare password.Here we will build a simple api fore register and login.when we will hash the password when user register and then compare that password with hash when they login. What is bcrypt ? bcrypt is a  password hashing function  designed by  Niels Provos  and David Mazières, based on the  Blowfish  cipher, and presented at  USENIX  in 1999.Besides incorporating a  salt  to protect against  rainbow table  attacks, bcrypt is an adaptive function: over time, the iteration count can be increased to make it slower, so it remains resistant to  brute-force search  attacks even with increasing computation power. What is hashing ?  Hashing is a one way function (well, a mapping). It's irreversible, you apply the secure hash algorithm and you cannot ge...
Read more

How to Upload Files Using AJAX in Node.js(Express)

Introduction In this blog, we will see how to upload an image file using AJAX. We will use the FileReader API to read the selected file, send the file data to the server via AJAX, and handle and save it as an image on the server side. Project Structure |--------- views |--------- demo.ejs |--------- app.js |--------- package.json Setup Folder Open your command prompt and create a new folder using the following command: mkdir upload Navigate into the newly created folder: cd upload Setup Node In Folder Initialize the Node.js environment by running: npm init -y This command creates a package.json file indicating that Node is initialized in the folder. Install Packages Install the required packages for building the application (Express, Body-Parser, EJS, and jQuery): npm install body-parser ejs express jquery Add Folders and Views Create a new folder named views in your project root, and inside it, create a file na...
Read more